{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业说明\n",
    "在Pima Indians Diabetes Data Set（皮马印第安人糖尿病数据集）进行分类器练习，数据集只有一个文件（diabetes.csv）：Pima Indians Diabetes Dataset 包括根据医疗记录的比马印第安人5 年内糖尿病的发病情况，这是一个两类分类问题。每个类的样本数目数量不均等。一共有 768 个样本，每个样本有8 个输入变量和1 个输出变量。缺失值通常用零值编码。\n",
    "\n",
    "1) 字段说明  \n",
    "    Pregnancies： 怀孕次数  \n",
    "    Glucose： 口服葡萄糖耐受试验中，2 小时的血浆葡萄糖浓度。  \n",
    "    BloodPressure： 舒张压（mm Hg）  \n",
    "    SkinThickness： 三头肌皮肤褶层厚度（mm）  \n",
    "    Insulin：2 小时血清胰岛素含量（μU/ ml）  \n",
    "    BMI： 体重指数（体重，kg /（身高，m）^ 2）  \n",
    "2) DiabetesPedigreeFunction： 糖尿病家族史  \n",
    "3) Age： 年龄（岁）  \n",
    "    Outcome： 输出变了/类别标签（0 或 1，出现糖尿病为1, 否则为 0）  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入必要工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "#竞赛的评价指标为logloss\n",
    "#from sklearn.metrics import log_loss  \n",
    "#SVM并不能直接输出各类的概率，所以在这个例子中我们用正确率作为模型预测性能的度量\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据 & 数据探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "data = pd.read_csv(\"diabetes.csv\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 768 entries, 0 to 767\n",
      "Data columns (total 9 columns):\n",
      "Pregnancies                 768 non-null int64\n",
      "Glucose                     768 non-null int64\n",
      "BloodPressure               768 non-null int64\n",
      "SkinThickness               768 non-null int64\n",
      "Insulin                     768 non-null int64\n",
      "BMI                         768 non-null float64\n",
      "DiabetesPedigreeFunction    768 non-null float64\n",
      "Age                         768 non-null int64\n",
      "Outcome                     768 non-null int64\n",
      "dtypes: float64(2), int64(7)\n",
      "memory usage: 54.1 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pregnancies     Glucose  BloodPressure  SkinThickness     Insulin  \\\n",
       "count   768.000000  768.000000     768.000000     768.000000  768.000000   \n",
       "mean      3.845052  120.894531      69.105469      20.536458   79.799479   \n",
       "std       3.369578   31.972618      19.355807      15.952218  115.244002   \n",
       "min       0.000000    0.000000       0.000000       0.000000    0.000000   \n",
       "25%       1.000000   99.000000      62.000000       0.000000    0.000000   \n",
       "50%       3.000000  117.000000      72.000000      23.000000   30.500000   \n",
       "75%       6.000000  140.250000      80.000000      32.000000  127.250000   \n",
       "max      17.000000  199.000000     122.000000      99.000000  846.000000   \n",
       "\n",
       "              BMI  DiabetesPedigreeFunction         Age     Outcome  \n",
       "count  768.000000                768.000000  768.000000  768.000000  \n",
       "mean    31.992578                  0.471876   33.240885    0.348958  \n",
       "std      7.884160                  0.331329   11.760232    0.476951  \n",
       "min      0.000000                  0.078000   21.000000    0.000000  \n",
       "25%     27.300000                  0.243750   24.000000    0.000000  \n",
       "50%     32.000000                  0.372500   29.000000    0.000000  \n",
       "75%     36.600000                  0.626250   41.000000    1.000000  \n",
       "max     67.100000                  2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 各属性统计特征\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从数据的大致取值可以看出，数据没有做脱敏处理，有单位和具体的物理含义，train.info虽然显示没有缺失值，但有些特征数据明显不对，如舒张压和体重等特征都不应该存在零值，（舒张压为0是说明这个人已经挂了么，体重身高比为0[捂脸]）应该是存在缺失值，但直接使用了零值进行填充，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAE+JJREFUeJzt3X+0ZWV93/H3BwbUoHH4MbIow2SwjmnJUpGMFEOrEW0q2jJIxcSyZIKzOsmqURLS1NGkksTYaCJiqC3trGAYLIUgFRmUxkxHwBUTKDMoIKJlpARmYZkB+aUULPDtH+e5cBj23LtnhnPP4d73a62zzt7Pefa53+sa7se9n72fJ1WFJEk72mvcBUiSJpMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp04JxF7AnDjrooFq6dOm4y5Ck55XNmzffW1WLZur3vA6IpUuXsmnTpnGXIUnPK0n+tk8/LzFJkjoZEJKkTgaEJKmTASFJ6mRASJI6jTQgktyR5OYk30yyqbUdkGRDktva+/6tPUnOSbIlyU1JjhplbZKk6c3GGcSbqurIqlre9tcAG6tqGbCx7QMcDyxrr9XAubNQmyRpJ8ZxiWkFsK5trwNOHGq/oAauBRYmOWQM9UmSGH1AFPCXSTYnWd3aDq6q7wO095e19kOBu4aO3draJEljMOonqY+tqruTvAzYkOQ70/RNR1s9q9MgaFYDLFmyZI8L/NnfumCPv0Nzz+Y/PnXcJUhjN9IziKq6u71vAy4Djgbumbp01N63te5bgcOGDl8M3N3xnWuranlVLV+0aMapRCRJu2lkAZFkvyQvmdoGfgH4FrAeWNm6rQQub9vrgVPb3UzHAA9OXYqSJM2+UV5iOhi4LMnUz/mvVfUXSa4HLkmyCrgTOLn1vxJ4G7AFeAQ4bYS1SZJmMLKAqKrbgdd0tN8HvLmjvYD3jaoeSdKu8UlqSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdRh4QSfZO8o0kX2r7hye5LsltSf48yb6t/QVtf0v7fOmoa5Mk7dxsnEGcDtw6tP8J4OyqWgbcD6xq7auA+6vqFcDZrZ8kaUxGGhBJFgNvB/607Qc4Dri0dVkHnNi2V7R92udvbv0lSWMw6jOITwP/Bniy7R8IPFBVj7f9rcChbftQ4C6A9vmDrb8kaQxmDIgk+yXZq22/MskJSfbpcdw/BbZV1ebh5o6u1eOz4e9dnWRTkk3bt2+fqQxJ0m7qcwbxNeCFSQ4FNgKnAef3OO5Y4IQkdwAXM7i09GlgYZIFrc9i4O62vRU4DKB9/lLgBzt+aVWtrarlVbV80aJFPcqQJO2OPgGRqnoEOAn491X1DuCImQ6qqg9V1eKqWgr8EvDVqjoFuAp4Z+u2Eri8ba9v+7TPv1pVzzqDkCTNjl4BkeT1wCnAl1vbgmn6z+SDwBlJtjAYYzivtZ8HHNjazwDW7MHPkCTtoT5/6H8d+BBwWVXdkuTlDM4Cequqq4Gr2/btwNEdfR4FTt6V75Ukjc6MAVFV1wDXJNmv7d8OfGDUhUmSxqvPXUyvT/Jt2sNuSV6T5D+OvDJJ0lj1GYP4NPBPgPsAqupG4A2jLEqSNH69HpSrqrt2aHpiBLVIkiZIn0Hqu5L8HFBtYr0P8My5lSRJc1CfM4hfBd7HYCqMrcCRbV+SNIf1uYvpXgbPQEiS5pE+dzGtS7JwaH//JJ8dbVmSpHHrc4np1VX1wNROVd0PvHZ0JUmSJkGfgNgryf5TO0kOYM+m2pAkPQ/0+UN/FvDXSaYW+TkZ+NjoSpIkTYI+g9QXJNkMvInBmg0nVdW3R16ZJGms+l4q+g6D9aMXACRZUlV3jqwqSdLYzRgQSd4PnAncw+AJ6jBY6e3Voy1NkjROfc4gTgd+uqruG3UxkqTJ0ecupruAB0ddiCRpsvQ5g7gduDrJl4HHphqr6lMjq0qSNHZ9AuLO9tq3vSRJ80Cf21x/DyDJflX1o9GXJEmaBK4oJ0nq5IpykqROrignSerkinKSpE6uKCdJ6jTtGUSSvYH3VJUryknSPDPtGURVPQGsmKVaJEkTpM8YxNeTfAb4c+Cp5yCq6oaRVSVJGrs+AfFz7f33h9oKOO65L0eSNClmGoPYCzi3qi6ZpXokSRNipjGIJ4Ffm6VaJEkTpM9trhuS/OskhyU5YOo18sokSWPVZwzive19+NmHAl7+3JcjSZoUfWZzPXw2CpEkTZY+a1Kf2tVeVRfMcNwLga8BL2g/59KqOjPJ4cDFwAHADQwexPtxkhcAFwA/y2BiwF+sqjt24XeRJD2H+oxBvG7o9Y+A3wVO6HHcY8BxVfUaBtNzvDXJMcAngLOrahlwP7Cq9V8F3F9VrwDObv0kSWPS5xLT+4f3k7wU+FyP4wr4Ydvdp72mnp/4F619HYPAOZfBE9u/29ovBT6TJO17JEmzrNd03zt4BFjWp2OSvZN8E9gGbAC+BzxQVY+3LlsZTAJIe78LoH3+IHDgbtQnSXoO9BmDuILB//OHQaAcAfR6cK7N5XRkkoXAZcDf7+o29aOm+Wy4ntXAaoAlS5b0KUOStBv63Ob6yaHtx4G/raqtu/JDquqBJFcDxwALkyxoZwmLgbtbt63AYcDWJAuAlwI/6PiutcBagOXLl3v5SZJGpM8lpjuB66rqmqr6OnBfkqUzHZRkUTtzIMmLgLcwWGjoKuCdrdtK4PK2vb7t0z7/quMPkjQ+fQLi88CTQ/tPtLaZHAJcleQm4HpgQ1V9CfggcEaSLQzGGM5r/c8DDmztZwBr+v0KkqRR6HOJaUFV/Xhqpz2zsO9MB1XVTcBrO9pvB47uaH8UOLlHPZKkWdDnDGJ7kqeee0iyArh3dCVJkiZBnzOIXwUubIsGwWAwufPpaknS3NHnQbnvAcckeTGQqnp49GVJksZtxktMSf5dkoVV9cOqejjJ/kn+YDaKkySNT58xiOOr6oGpnaq6H3jb6EqSJE2CPgGxd5tpFXjqmYYXTNNfkjQH9Bmk/i/AxiR/xmDqi/cymGRPkjSH9Rmk/qP2sNtbWtNHq+oroy1LkjRufc4gAL7B09N1f2N05Uiacufvv2rcJWgCLfnIzbP2s/rcxfQu4H8ymB/pXcB1Sd45/VGSpOe7PmcQvw28rqq2wWASPuB/MFjUR5I0R/W5i2mvqXBo7ut5nCTpeazPGcRfJPkKcFHb/0XgytGVJEmaBH3uYvqtJCcB/5DBqm9rq+qykVcmSRqrXncxVdUXgC+MuBZJ0gRxLEGS1MmAkCR12mlAJNnY3j8xe+VIkibFdGMQhyR5I3BCkosZDFA/papuGGllkqSxmi4gPgKsARYDn9rhswKOG1VRkqTx22lAVNWlwKVJ/m1VfXQWa5IkTYA+z0F8NMkJwBta09VV9aXRliVJGrc+k/X9IXA68O32Or21SZLmsD4Pyr0dOLKqngRIso7BlN8fGmVhkqTx6vscxMKh7ZeOohBJ0mTpcwbxh8A3klzF4FbXN+DZgyTNeX0GqS9KcjXwOgYB8cGq+j+jLkySNF59J+v7PrB+xLVIkiaIczFJkjoZEJKkTtMGRJK9knxrtoqRJE2OaQOiPftwY5Ils1SPJGlC9LnEdAhwS5KNSdZPvWY6KMlhSa5KcmuSW5Kc3toPSLIhyW3tff/WniTnJNmS5KYkR+3ZryZJ2hN97mL6vd387seB36yqG5K8BNicZAPwy8DGqvp4kjUMZoz9IHA8sKy9/gFwbnuXJI3BjGcQVXUNcAewT9u+HphxLYiq+v7UmhFV9TBwK3AosAJY17qtA05s2yuAC2rgWmBhkkN27deRJD1X+kzW9y+BS4H/3JoOBb64Kz8kyVLgtcB1wMHtuYqp5yteNvS9dw0dtrW1SZLGoM8YxPuAY4GHAKrqNp7+oz6jJC8G/hvw61X10HRdO9qq4/tWJ9mUZNP27dv7liFJ2kV9AuKxqvrx1E6SBXT84e6SZB8G4XBhVX2hNd8zdemovW9r7VuBw4YOXwzcveN3VtXaqlpeVcsXLVrUpwxJ0m7oExDXJPkw8KIk/xj4PHDFTAclCXAecGtVDS9Zuh5Y2bZXApcPtZ/a7mY6Bnhw6lKUJGn29bmLaQ2wCrgZ+BXgSuBPexx3LPAe4OYk32xtHwY+DlySZBVwJ3By++xK4G3AFuAR4LSev4MkaQT6zOb6ZFsk6DoGl5a+W1UzXmKqqr+ie1wB4M0d/YvBeIckaQLMGBBJ3g78J+B7DP7gH57kV6rqv4+6OEnS+PS5xHQW8Kaq2gKQ5O8CXwYMCEmaw/oMUm+bCofmdp6+80iSNEft9AwiyUlt85YkVwKXMBiDOJnB09SSpDlsuktM/2xo+x7gjW17O7D/yCqSJE2EnQZEVXmbqSTNY33uYjoceD+wdLh/VZ0wurIkSePW5y6mLzJ4IvoK4MnRliNJmhR9AuLRqjpn5JVIkiZKn4D4kyRnAn8JPDbVOLXWgyRpbuoTEK9iMKfScTx9ianaviRpjuoTEO8AXj485bckae7r8yT1jcDCURciSZosfc4gDga+k+R6njkG4W2ukjSH9QmIM0dehSRp4vRZD+Ka2ShEkjRZ+jxJ/TBPr0G9L7AP8KOq+slRFiZJGq8+ZxAvGd5PciJw9MgqkiRNhD53MT1DVX0Rn4GQpDmvzyWmk4Z29wKW8/QlJ0nSHNXnLqbhdSEeB+4AVoykGknSxOgzBuG6EJI0D0235OhHpjmuquqjI6hHkjQhpjuD+FFH237AKuBAwICQpDlsuiVHz5raTvIS4HTgNOBi4KydHSdJmhumHYNIcgBwBnAKsA44qqrun43CJEnjNd0YxB8DJwFrgVdV1Q9nrSpJ0thN96DcbwJ/B/gd4O4kD7XXw0kemp3yJEnjMt0YxC4/ZS1JmjsMAUlSJwNCktTJgJAkdTIgJEmdRhYQST6bZFuSbw21HZBkQ5Lb2vv+rT1JzkmyJclNSY4aVV2SpH5GeQZxPvDWHdrWABurahmwse0DHA8sa6/VwLkjrEuS1MPIAqKqvgb8YIfmFQyeyKa9nzjUfkENXAssTHLIqGqTJM1stscgDq6q7wO095e19kOBu4b6bW1tz5JkdZJNSTZt3759pMVK0nw2KYPU6WjrXLWuqtZW1fKqWr5o0aIRlyVJ89dsB8Q9U5eO2vu21r4VOGyo32Lg7lmuTZI0ZLYDYj2wsm2vBC4faj+13c10DPDg1KUoSdJ49FmTerckuQj4eeCgJFuBM4GPA5ckWQXcCZzcul8JvA3YAjzCYN0JSdIYjSwgqurdO/nozR19C3jfqGqRJO26SRmkliRNGANCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnSYqIJK8Ncl3k2xJsmbc9UjSfDYxAZFkb+A/AMcDRwDvTnLEeKuSpPlrYgICOBrYUlW3V9WPgYuBFWOuSZLmrUkKiEOBu4b2t7Y2SdIYLBh3AUPS0VbP6pSsBla33R8m+e5Iq5pfDgLuHXcRkyCfXDnuEvRM/tuccmbXn8pd9lN9Ok1SQGwFDhvaXwzcvWOnqloLrJ2touaTJJuqavm465B25L/N8ZikS0zXA8uSHJ5kX+CXgPVjrkmS5q2JOYOoqseT/BrwFWBv4LNVdcuYy5KkeWtiAgKgqq4Erhx3HfOYl+40qfy3OQapetY4sCRJEzUGIUmaIAaEnOJEEyvJZ5NsS/KtcdcyHxkQ85xTnGjCnQ+8ddxFzFcGhJziRBOrqr4G/GDcdcxXBoSc4kRSJwNCvaY4kTT/GBDqNcWJpPnHgJBTnEjqZEDMc1X1ODA1xcmtwCVOcaJJkeQi4G+An06yNcmqcdc0n/gktSSpk2cQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaE5r0ki5NcnuS2JN9L8iftmZDpjvnwbNUnjYsBoXktSYAvAF+sqmXAK4EXAx+b4VADQnOeAaH57jjg0ar6M4CqegL4DeC9Sf5Vks9MdUzypSQ/n+TjwIuSfDPJhe2zU5PclOTGJJ9rbT+VZGNr35hkSWs/P8m5Sa5KcnuSN7Z1D25Ncv7Qz/uFJH+T5IYkn0/y4ln7X0XCgJB+Btg83FBVDwF3spM126tqDfB/q+rIqjolyc8Avw0cV1WvAU5vXT8DXFBVrwYuBM4Z+pr9GYTTbwBXAGe3Wl6V5MgkBwG/A7ylqo4CNgFnPBe/sNRX538A0jwSumev3Vl7l+OAS6vqXoCqmlq/4PXASW37c8AfDR1zRVVVkpuBe6rqZoAktwBLGUyaeATw9cFVMPZlMOWENGsMCM13twD/fLghyU8ymOH2QZ55lv3CnXxH3zAZ7vNYe39yaHtqfwHwBLChqt7d43ulkfASk+a7jcBPJDkVnlqC9SwGS13eDhyZZK8khzFYfW/K/0uyz9B3vCvJge07Dmjtf81gdlyAU4C/2oW6rgWOTfKK9p0/keSVu/rLSXvCgNC8VoPZKt8BnJzkNuB/AY8yuEvp68D/Bm4GPgncMHToWuCmJBe22W8/BlyT5EbgU63PB4DTktwEvIenxyb61LUd+GXgonb8tcDf293fU9odzuYqSerkGYQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE7/H6faJrOoDn8hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Target 分布，看看各类样本分布是否均衡\n",
    "sns.countplot(data.Outcome);\n",
    "pyplot.xlabel('Outcome');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分类结果中正例的样本约为反例样本的一半多一些，样本不均衡，交叉验证对分类任务缺省的是采用StratifiedKFold，在每折采样时根据各类样本按比例采样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据准备"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768, 7)"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data.drop('SkinThickness', axis=1)\n",
    "data = data.drop('Insulin', axis=1)\n",
    "\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "三头肌皮肤褶层厚度和血清胰岛素两个特征为0值的样本数太多，但不确定是否去除之后更合理些，通过后续模型训练，分别尝试去除看效果如何"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768, 7)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for i in range(768):\n",
    "    if data.Glucose.values[i] == 0:\n",
    "        data.Glucose.values[i] = 117\n",
    "    if data.BloodPressure.values[i] == 0:\n",
    "        data.BloodPressure.values[i] = 72\n",
    "    if data.BMI.values[i] == 0:\n",
    "        data.BMI.values[i] = 32\n",
    "        \n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从特征工程可看出，有三个特征（血浆葡萄糖浓度、舒张压、体重指数）对应的样本数据存在少部分缺失，其中零值即为缺失值，使用对应特征的中位数进行替换填充，由于读取的数据文件缺失值使用了零值编码，而不是直接空着，无法使用sklearn.preprocessing.Imputer进行填充，故使用循环判断的方式进行填充 \n",
    "\n",
    "其中中位数如下：  \n",
    "Glucose ：117  \n",
    "BloodPressure ：72  \n",
    "BMI ：32  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从原始数据中分离输入特征x和输出y\n",
    "y = data['Outcome'].values\n",
    "X = data.drop('Outcome', axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(614, 6)"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#将数据分割训练数据与测试数据\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 随机采样20%的数据构建测试样本，其余作为训练样本\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=33, test_size=0.2)\n",
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = np.array(X_train)\n",
    "X_test = np.array(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据标准化\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 初始化特征的标准化器\n",
    "ss_X = StandardScaler()\n",
    "\n",
    "# 分别对训练和测试数据的特征进行标准化处理\n",
    "X_train = ss_X.fit_transform(X_train)\n",
    "X_test = ss_X.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SVM（训练中增加权重参数，后续解释）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LinearSVC_performance(C, X_train, y_train, X_test, y_test):\n",
    "    \n",
    "    # 在训练集是那个利用SVC训练\n",
    "    SVC2 =  LinearSVC(C = C, class_weight={0:0.3489, 1:0.6511})\n",
    "    SVC2 = SVC2.fit(X_train, y_train)\n",
    "    \n",
    "    # 在校验集上返回accuracy\n",
    "    accuracy = SVC2.score(X_test, y_test)\n",
    "    \n",
    "    print(\"accuracy: {}\".format(accuracy))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义模型性能最终评估函数，在测试集上使用训练得到的LinearSVC（包含已确定的参数C）进行数据分类，并返回正确率"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1 default SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于LinerSVC是基于libliner实现的SVM的，而SVC/NuSVC是基于libsvm实现的，所以如果核函数想采用的线性核，linerSVC的训练效率更高一点，且可以有更多正则选择（L1/L2）和损失函数选择"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "SVC1_train = LinearSVC(class_weight={0:0.3489, 1:0.6511})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为数据探索时发现，样本在分类时，0类别比1类别多很多，所以在训练时，加上class_weight权重，让模型训练更合理些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy of each fold is:  [0.76612903 0.72357724 0.7804878  0.81967213 0.71311475]\n",
      "cv accuracy is: 0.7605961916308746\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import cross_val_score\n",
    "accuracy = cross_val_score(SVC1_train, X_train, y_train, cv=5, scoring='accuracy')\n",
    "print('accuracy of each fold is: ',accuracy)\n",
    "print('cv accuracy is:', accuracy.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于本数据集的样本数只有768个，数据量不大，使用交叉验证得出模型在训练集上的正确率比较合适  \n",
    "上述结果得出，缺省参数（C = 1.0）的LinearSVC模型训练后，在训练集上的准确率为 0.7605961916308746"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for classifier LinearSVC(C=1.0, class_weight={0: 0.3489, 1: 0.6511}, dual=True,\n",
      "     fit_intercept=True, intercept_scaling=1, loss='squared_hinge',\n",
      "     max_iter=1000, multi_class='ovr', penalty='l2', random_state=None,\n",
      "     tol=0.0001, verbose=0):\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.80      0.83      0.82        99\n",
      "          1       0.67      0.64      0.65        55\n",
      "\n",
      "avg / total       0.76      0.76      0.76       154\n",
      "\n",
      "\n",
      "Confusion matrix:\n",
      "[[82 17]\n",
      " [20 35]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "#使用训练集训练出模型，并在测试集上看模型预测的情况\n",
    "SVC1 = LinearSVC(class_weight={0:0.3489, 1:0.6511}).fit(X_train, y_train)\n",
    "y_predict = SVC1.predict(X_test)\n",
    "\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (SVC1, classification_report(y_test, y_predict)))\n",
    "print(\"Confusion matrix:\\n%s\" % confusion_matrix(y_test, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对角线参数表示分正确的数目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7597402597402597\n",
      "default SVC accuracy on test data: 0.7597402597402597\n"
     ]
    }
   ],
   "source": [
    "default_SVC_accuracy = LinearSVC_performance(1.0, X_train, y_train, X_test, y_test)\n",
    "print('default SVC accuracy on test data:', default_SVC_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "缺省参数（C = 1.0）的LinearSVC模型训练后，在测试集上的准确率为 0.7597402597402597，比训练集略低一点点"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 线性SVM正则参数调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LinearSVC(C=1.0, class_weight={0: 0.3489, 1: 0.6511}, dual=True,\n",
       "     fit_intercept=True, intercept_scaling=1, loss='squared_hinge',\n",
       "     max_iter=1000, multi_class='ovr', penalty='l2', random_state=None,\n",
       "     tol=0.0001, verbose=0),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'C': array([1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "#Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#penaltys = ['l1','l2']\n",
    "Cs = np.logspace(-3, 3, 7)\n",
    "tuned_parameters = dict(C = Cs)\n",
    "\n",
    "SVC3 = LinearSVC(class_weight={0:0.3489, 1:0.6511})\n",
    "grid= GridSearchCV(SVC3, tuned_parameters,cv=5, scoring='accuracy')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为样本数量不大，这边直接使用GridSearchCV对不同候选参数C进行训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.00100074, 0.00140028, 0.00360222, 0.01340714, 0.03602319,\n",
       "        0.03502364, 0.03602395]),\n",
       " 'mean_score_time': array([0.00060081, 0.00080099, 0.00040035, 0.00020051, 0.00020008,\n",
       "        0.00020037, 0.00040026]),\n",
       " 'mean_test_score': array([0.72638436, 0.76710098, 0.75895765, 0.76058632, 0.76058632,\n",
       "        0.75081433, 0.6514658 ]),\n",
       " 'mean_train_score': array([0.72597855, 0.75774223, 0.76670024, 0.76792224, 0.7675149 ,\n",
       "        0.76872528, 0.64939249]),\n",
       " 'param_C': masked_array(data=[0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0],\n",
       "              mask=[False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.001},\n",
       "  {'C': 0.01},\n",
       "  {'C': 0.1},\n",
       "  {'C': 1.0},\n",
       "  {'C': 10.0},\n",
       "  {'C': 100.0},\n",
       "  {'C': 1000.0}],\n",
       " 'rank_test_score': array([6, 1, 4, 2, 2, 5, 7]),\n",
       " 'split0_test_score': array([0.74193548, 0.75806452, 0.76612903, 0.76612903, 0.76612903,\n",
       "        0.74193548, 0.56451613]),\n",
       " 'split0_train_score': array([0.7244898 , 0.76326531, 0.7755102 , 0.7755102 , 0.7755102 ,\n",
       "        0.75918367, 0.60612245]),\n",
       " 'split1_test_score': array([0.71544715, 0.73170732, 0.71544715, 0.72357724, 0.72357724,\n",
       "        0.73170732, 0.74796748]),\n",
       " 'split1_train_score': array([0.72505092, 0.75356415, 0.75763747, 0.76171079, 0.76171079,\n",
       "        0.77393075, 0.74949084]),\n",
       " 'split2_test_score': array([0.71544715, 0.78861789, 0.7804878 , 0.7804878 , 0.7804878 ,\n",
       "        0.77235772, 0.58536585]),\n",
       " 'split2_train_score': array([0.73319756, 0.76578411, 0.77393075, 0.77596741, 0.77393075,\n",
       "        0.77189409, 0.54378819]),\n",
       " 'split3_test_score': array([0.77868852, 0.82786885, 0.81967213, 0.81967213, 0.81967213,\n",
       "        0.81967213, 0.68852459]),\n",
       " 'split3_train_score': array([0.71138211, 0.74593496, 0.7601626 , 0.75813008, 0.75813008,\n",
       "        0.77642276, 0.69105691]),\n",
       " 'split4_test_score': array([0.68032787, 0.7295082 , 0.71311475, 0.71311475, 0.71311475,\n",
       "        0.68852459, 0.67213115]),\n",
       " 'split4_train_score': array([0.73577236, 0.7601626 , 0.76626016, 0.76829268, 0.76829268,\n",
       "        0.76219512, 0.65650407]),\n",
       " 'std_fit_time': array([5.56082906e-07, 4.89745803e-04, 8.00943410e-04, 2.72922185e-03,\n",
       "        2.53137287e-03, 1.09658523e-03, 1.99945010e-03]),\n",
       " 'std_score_time': array([0.00049056, 0.0004005 , 0.00049033, 0.00040102, 0.00040016,\n",
       "        0.00040073, 0.00049021]),\n",
       " 'std_test_score': array([0.03258564, 0.03707729, 0.04040444, 0.03874764, 0.03874764,\n",
       "        0.0435325 , 0.0680101 ]),\n",
       " 'std_train_score': array([0.00853423, 0.00718196, 0.00714092, 0.00716809, 0.00674471,\n",
       "        0.00678343, 0.07047091])}"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7671009771986971\n",
      "{'C': 0.01}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7662337662337663\n",
      "default SVC accuracy on test data: 0.7662337662337663\n"
     ]
    }
   ],
   "source": [
    "best_SVC_accuracy = LinearSVC_performance(0.01, X_train, y_train, X_test, y_test)\n",
    "print('default SVC accuracy on test data:', best_SVC_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现最佳的参数 C 最优取值是 0.01，此时在训练集上得到的正确率是0.757328990228013，和上面直接做缺省参数LinearSVC得到的准确率差不多  \n",
    "第一次 缺省参数LinearSVC 训练集上正确率：0.7605961916308746  测试集上正确率：0.7597402597402597  \n",
    "第二次 参数调优LinearSVC 训练集上正确率：0.7671009771986971  测试集上正确率： 0.7662337662337663  \n",
    "\n",
    "在选出最优参数C=0.01后，训练的效果比默认参数C=1好，在训练集和测试集上的正确率均有所提升"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 RBF核SVM正则参数调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "def SVC_performance(C, gamma, X_train, y_train, X_test, y_test):\n",
    "    \n",
    "    # 在训练集是那个利用SVC训练\n",
    "    SVC4 =  SVC(C = C, kernel='rbf', gamma = gamma, class_weight={0:0.3489, 1:0.6511})\n",
    "    SVC4 = SVC4.fit(X_train, y_train)\n",
    "    \n",
    "    # 在校验集上返回accuracy\n",
    "    accuracy = SVC4.score(X_test, y_test)\n",
    "    \n",
    "    print(\"accuracy: {}\".format(accuracy))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义模型性能最终评估函数，在测试集上使用训练得到的SVC（包含最优参数C，gamma）进行数据分类，并返回正确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=SVC(C=1.0, cache_size=200, class_weight={0: 0.3489, 1: 0.6511}, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'gamma': array([1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03,\n",
       "       1.e+04]), 'C': array([1.e-04, 1.e-03, 1.e-02, 1.e-01, 1.e+00, 1.e+01, 1.e+02, 1.e+03,\n",
       "       1.e+04])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='accuracy', verbose=0)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "C_s = np.logspace(-4, 4, 9)\n",
    "gamma_s = np.logspace(-4, 4, 9) \n",
    "#C_s = [0.001, 0.01, 0.1, 1, 10, 100, 1000, 10000]\n",
    "#gamma_s = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(gamma = gamma_s, C = C_s)\n",
    "\n",
    "SVC5 = SVC(class_weight={0:0.3489, 1:0.6511})\n",
    "grid= GridSearchCV(SVC5, tuned_parameters,cv=5, scoring='accuracy')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.01560941, 0.0130085 , 0.0130084 , 0.01280861, 0.01300793,\n",
       "        0.0172122 , 0.02161441, 0.02381539, 0.02321548, 0.01300859,\n",
       "        0.01300845, 0.01300802, 0.01300921, 0.01300864, 0.01701083,\n",
       "        0.02201481, 0.02341633, 0.02301574, 0.01300864, 0.01300735,\n",
       "        0.01300879, 0.01300869, 0.01300864, 0.01701045, 0.02181411,\n",
       "        0.02341518, 0.02341523, 0.01300864, 0.0128087 , 0.01300859,\n",
       "        0.01160698, 0.01300821, 0.01801271, 0.02261524, 0.02381592,\n",
       "        0.02381601, 0.01300874, 0.01280861, 0.01000681, 0.0092062 ,\n",
       "        0.01220822, 0.01761179, 0.02381577, 0.02501645, 0.02521672,\n",
       "        0.01300845, 0.00960536, 0.00920587, 0.01000686, 0.01581025,\n",
       "        0.02641754, 0.02881894, 0.02901859, 0.02901902, 0.00980668,\n",
       "        0.00940585, 0.01020751, 0.0198132 , 0.01781154, 0.02981882,\n",
       "        0.02801809, 0.02901864, 0.02941909, 0.00960693, 0.01100736,\n",
       "        0.02661762, 0.09546256, 0.01961188, 0.02801681, 0.02961969,\n",
       "        0.03122101, 0.02961912, 0.01120701, 0.02941928, 0.16651058,\n",
       "        0.38766069, 0.01761179, 0.02641749, 0.0282187 , 0.02901931,\n",
       "        0.02881832]),\n",
       " 'mean_score_time': array([0.00340261, 0.00240107, 0.00260262, 0.00260172, 0.00260162,\n",
       "        0.00240154, 0.0050034 , 0.00560384, 0.00540366, 0.00260172,\n",
       "        0.00240149, 0.00260153, 0.00260158, 0.00240145, 0.00280166,\n",
       "        0.00480299, 0.00520349, 0.00560384, 0.00240169, 0.00240164,\n",
       "        0.00260153, 0.0028018 , 0.00240169, 0.00280161, 0.00480304,\n",
       "        0.00520325, 0.00520334, 0.00240145, 0.00280175, 0.00240145,\n",
       "        0.00220122, 0.00260153, 0.00200119, 0.00440278, 0.00560379,\n",
       "        0.00540361, 0.0028019 , 0.00260153, 0.00200124, 0.00180106,\n",
       "        0.00220127, 0.00240173, 0.00480294, 0.00580397, 0.00540333,\n",
       "        0.0024014 , 0.00200157, 0.00200138, 0.00160136, 0.00200143,\n",
       "        0.00280185, 0.00480285, 0.00520291, 0.0052032 , 0.00160065,\n",
       "        0.00160136, 0.00180116, 0.00160117, 0.00200167, 0.00240197,\n",
       "        0.00500302, 0.00520353, 0.0052031 , 0.00180106, 0.0018012 ,\n",
       "        0.00140104, 0.00160127, 0.00200109, 0.00260177, 0.00480318,\n",
       "        0.00640454, 0.00580544, 0.00180163, 0.00160146, 0.00160151,\n",
       "        0.00120134, 0.00200143, 0.00260162, 0.00500336, 0.00500317,\n",
       "        0.00560369]),\n",
       " 'mean_test_score': array([0.65309446, 0.65309446, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.65309446, 0.65309446, 0.76221498,\n",
       "        0.71986971, 0.66286645, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.76547231, 0.752443  , 0.76221498,\n",
       "        0.68729642, 0.65309446, 0.65309446, 0.65309446, 0.65309446,\n",
       "        0.76221498, 0.75407166, 0.76872964, 0.75407166, 0.72801303,\n",
       "        0.65472313, 0.65309446, 0.65309446, 0.65309446, 0.75570033,\n",
       "        0.76384365, 0.76547231, 0.73452769, 0.72149837, 0.65472313,\n",
       "        0.65309446, 0.65309446, 0.65309446, 0.76221498, 0.76384365,\n",
       "        0.75732899, 0.67915309, 0.71986971, 0.65472313, 0.65309446,\n",
       "        0.65309446, 0.65309446, 0.76384365, 0.76221498, 0.752443  ,\n",
       "        0.66123779, 0.71986971, 0.65472313, 0.65309446, 0.65309446,\n",
       "        0.65309446]),\n",
       " 'mean_train_score': array([0.65309498, 0.65309498, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.65309498, 0.65309498, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.65309498, 0.65309498, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.65309498, 0.65309498, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.65309498, 0.65309498, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.65309498, 0.65309498, 0.65309498, 0.65309498, 0.76710259,\n",
       "        0.72434755, 0.70596704, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.65309498, 0.65309498, 0.76100752, 0.75815039, 0.78582913,\n",
       "        0.84163374, 0.99348102, 0.65309498, 0.65309498, 0.65309498,\n",
       "        0.76182467, 0.7601829 , 0.77484109, 0.83021846, 0.98819315,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.76018456,\n",
       "        0.76873772, 0.78135344, 0.87133918, 1.        , 1.        ,\n",
       "        1.        , 1.        , 1.        , 0.76751904, 0.77077357,\n",
       "        0.80171926, 0.93037426, 1.        , 1.        , 1.        ,\n",
       "        1.        , 1.        , 0.77077273, 0.77850709, 0.83143299,\n",
       "        0.97678866, 1.        , 1.        , 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'param_C': masked_array(data=[0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001,\n",
       "                    0.0001, 0.0001, 0.001, 0.001, 0.001, 0.001, 0.001,\n",
       "                    0.001, 0.001, 0.001, 0.001, 0.01, 0.01, 0.01, 0.01,\n",
       "                    0.01, 0.01, 0.01, 0.01, 0.01, 0.1, 0.1, 0.1, 0.1, 0.1,\n",
       "                    0.1, 0.1, 0.1, 0.1, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0,\n",
       "                    1.0, 1.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0, 10.0,\n",
       "                    10.0, 10.0, 100.0, 100.0, 100.0, 100.0, 100.0, 100.0,\n",
       "                    100.0, 100.0, 100.0, 1000.0, 1000.0, 1000.0, 1000.0,\n",
       "                    1000.0, 1000.0, 1000.0, 1000.0, 1000.0, 10000.0,\n",
       "                    10000.0, 10000.0, 10000.0, 10000.0, 10000.0, 10000.0,\n",
       "                    10000.0, 10000.0],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_gamma': masked_array(data=[0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0,\n",
       "                    10000.0, 0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0,\n",
       "                    1000.0, 10000.0, 0.0001, 0.001, 0.01, 0.1, 1.0, 10.0,\n",
       "                    100.0, 1000.0, 10000.0, 0.0001, 0.001, 0.01, 0.1, 1.0,\n",
       "                    10.0, 100.0, 1000.0, 10000.0, 0.0001, 0.001, 0.01, 0.1,\n",
       "                    1.0, 10.0, 100.0, 1000.0, 10000.0, 0.0001, 0.001, 0.01,\n",
       "                    0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0, 0.0001, 0.001,\n",
       "                    0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0, 0.0001,\n",
       "                    0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0,\n",
       "                    0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0,\n",
       "                    10000.0],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.0001, 'gamma': 0.0001},\n",
       "  {'C': 0.0001, 'gamma': 0.001},\n",
       "  {'C': 0.0001, 'gamma': 0.01},\n",
       "  {'C': 0.0001, 'gamma': 0.1},\n",
       "  {'C': 0.0001, 'gamma': 1.0},\n",
       "  {'C': 0.0001, 'gamma': 10.0},\n",
       "  {'C': 0.0001, 'gamma': 100.0},\n",
       "  {'C': 0.0001, 'gamma': 1000.0},\n",
       "  {'C': 0.0001, 'gamma': 10000.0},\n",
       "  {'C': 0.001, 'gamma': 0.0001},\n",
       "  {'C': 0.001, 'gamma': 0.001},\n",
       "  {'C': 0.001, 'gamma': 0.01},\n",
       "  {'C': 0.001, 'gamma': 0.1},\n",
       "  {'C': 0.001, 'gamma': 1.0},\n",
       "  {'C': 0.001, 'gamma': 10.0},\n",
       "  {'C': 0.001, 'gamma': 100.0},\n",
       "  {'C': 0.001, 'gamma': 1000.0},\n",
       "  {'C': 0.001, 'gamma': 10000.0},\n",
       "  {'C': 0.01, 'gamma': 0.0001},\n",
       "  {'C': 0.01, 'gamma': 0.001},\n",
       "  {'C': 0.01, 'gamma': 0.01},\n",
       "  {'C': 0.01, 'gamma': 0.1},\n",
       "  {'C': 0.01, 'gamma': 1.0},\n",
       "  {'C': 0.01, 'gamma': 10.0},\n",
       "  {'C': 0.01, 'gamma': 100.0},\n",
       "  {'C': 0.01, 'gamma': 1000.0},\n",
       "  {'C': 0.01, 'gamma': 10000.0},\n",
       "  {'C': 0.1, 'gamma': 0.0001},\n",
       "  {'C': 0.1, 'gamma': 0.001},\n",
       "  {'C': 0.1, 'gamma': 0.01},\n",
       "  {'C': 0.1, 'gamma': 0.1},\n",
       "  {'C': 0.1, 'gamma': 1.0},\n",
       "  {'C': 0.1, 'gamma': 10.0},\n",
       "  {'C': 0.1, 'gamma': 100.0},\n",
       "  {'C': 0.1, 'gamma': 1000.0},\n",
       "  {'C': 0.1, 'gamma': 10000.0},\n",
       "  {'C': 1.0, 'gamma': 0.0001},\n",
       "  {'C': 1.0, 'gamma': 0.001},\n",
       "  {'C': 1.0, 'gamma': 0.01},\n",
       "  {'C': 1.0, 'gamma': 0.1},\n",
       "  {'C': 1.0, 'gamma': 1.0},\n",
       "  {'C': 1.0, 'gamma': 10.0},\n",
       "  {'C': 1.0, 'gamma': 100.0},\n",
       "  {'C': 1.0, 'gamma': 1000.0},\n",
       "  {'C': 1.0, 'gamma': 10000.0},\n",
       "  {'C': 10.0, 'gamma': 0.0001},\n",
       "  {'C': 10.0, 'gamma': 0.001},\n",
       "  {'C': 10.0, 'gamma': 0.01},\n",
       "  {'C': 10.0, 'gamma': 0.1},\n",
       "  {'C': 10.0, 'gamma': 1.0},\n",
       "  {'C': 10.0, 'gamma': 10.0},\n",
       "  {'C': 10.0, 'gamma': 100.0},\n",
       "  {'C': 10.0, 'gamma': 1000.0},\n",
       "  {'C': 10.0, 'gamma': 10000.0},\n",
       "  {'C': 100.0, 'gamma': 0.0001},\n",
       "  {'C': 100.0, 'gamma': 0.001},\n",
       "  {'C': 100.0, 'gamma': 0.01},\n",
       "  {'C': 100.0, 'gamma': 0.1},\n",
       "  {'C': 100.0, 'gamma': 1.0},\n",
       "  {'C': 100.0, 'gamma': 10.0},\n",
       "  {'C': 100.0, 'gamma': 100.0},\n",
       "  {'C': 100.0, 'gamma': 1000.0},\n",
       "  {'C': 100.0, 'gamma': 10000.0},\n",
       "  {'C': 1000.0, 'gamma': 0.0001},\n",
       "  {'C': 1000.0, 'gamma': 0.001},\n",
       "  {'C': 1000.0, 'gamma': 0.01},\n",
       "  {'C': 1000.0, 'gamma': 0.1},\n",
       "  {'C': 1000.0, 'gamma': 1.0},\n",
       "  {'C': 1000.0, 'gamma': 10.0},\n",
       "  {'C': 1000.0, 'gamma': 100.0},\n",
       "  {'C': 1000.0, 'gamma': 1000.0},\n",
       "  {'C': 1000.0, 'gamma': 10000.0},\n",
       "  {'C': 10000.0, 'gamma': 0.0001},\n",
       "  {'C': 10000.0, 'gamma': 0.001},\n",
       "  {'C': 10000.0, 'gamma': 0.01},\n",
       "  {'C': 10000.0, 'gamma': 0.1},\n",
       "  {'C': 10000.0, 'gamma': 1.0},\n",
       "  {'C': 10000.0, 'gamma': 10.0},\n",
       "  {'C': 10000.0, 'gamma': 100.0},\n",
       "  {'C': 10000.0, 'gamma': 1000.0},\n",
       "  {'C': 10000.0, 'gamma': 10000.0}],\n",
       " 'rank_test_score': array([32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,\n",
       "        32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32,  7, 21, 26, 32, 32,\n",
       "        32, 32, 32,  2, 16,  7, 24, 32, 32, 32, 32,  7, 14,  1, 14, 19, 28,\n",
       "        32, 32, 32, 13,  4,  2, 18, 20, 28, 32, 32, 32,  7,  4, 12, 25, 21,\n",
       "        28, 32, 32, 32,  4,  7, 16, 27, 21, 28, 32, 32, 32]),\n",
       " 'split0_test_score': array([0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.77419355,\n",
       "        0.74193548, 0.66129032, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.77419355, 0.73387097, 0.76612903,\n",
       "        0.67741935, 0.65322581, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.77419355, 0.73387097, 0.75806452, 0.76612903, 0.73387097,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.65322581, 0.74193548,\n",
       "        0.76612903, 0.76612903, 0.75      , 0.72580645, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.65322581, 0.75806452, 0.75806452,\n",
       "        0.77419355, 0.7016129 , 0.72580645, 0.65322581, 0.65322581,\n",
       "        0.65322581, 0.65322581, 0.77419355, 0.76612903, 0.75      ,\n",
       "        0.61290323, 0.72580645, 0.65322581, 0.65322581, 0.65322581,\n",
       "        0.65322581]),\n",
       " 'split0_train_score': array([0.65306122, 0.65306122, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.65306122, 0.65306122, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.65306122, 0.65306122, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.65306122, 0.65306122, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.65306122, 0.65306122, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.65306122, 0.65306122, 0.65306122, 0.65306122, 0.76530612,\n",
       "        0.71836735, 0.67346939, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.65306122, 0.65306122, 0.76938776, 0.76326531, 0.78367347,\n",
       "        0.86326531, 0.99387755, 0.65306122, 0.65306122, 0.65306122,\n",
       "        0.77142857, 0.76122449, 0.78163265, 0.83265306, 0.98979592,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.76326531,\n",
       "        0.77346939, 0.78571429, 0.87346939, 1.        , 1.        ,\n",
       "        1.        , 1.        , 1.        , 0.7755102 , 0.77959184,\n",
       "        0.81020408, 0.92653061, 1.        , 1.        , 1.        ,\n",
       "        1.        , 1.        , 0.7755102 , 0.7877551 , 0.82653061,\n",
       "        0.97346939, 1.        , 1.        , 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'split1_test_score': array([0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.71544715,\n",
       "        0.72357724, 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.71544715, 0.72357724, 0.74796748,\n",
       "        0.69105691, 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.71544715, 0.72357724, 0.74796748, 0.70731707, 0.72357724,\n",
       "        0.65853659, 0.6504065 , 0.6504065 , 0.6504065 , 0.72357724,\n",
       "        0.72357724, 0.74796748, 0.68292683, 0.74796748, 0.65853659,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.72357724, 0.7398374 ,\n",
       "        0.7398374 , 0.68292683, 0.7398374 , 0.65853659, 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.71544715, 0.7398374 , 0.70731707,\n",
       "        0.65853659, 0.7398374 , 0.65853659, 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 ]),\n",
       " 'split1_train_score': array([0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.77393075,\n",
       "        0.71894094, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.76985743, 0.75967413, 0.79022403,\n",
       "        0.83910387, 0.98778004, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.77393075, 0.76374745, 0.76985743, 0.84928717, 0.98574338,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.76374745,\n",
       "        0.77393075, 0.77596741, 0.87983707, 1.        , 1.        ,\n",
       "        1.        , 1.        , 1.        , 0.77393075, 0.76578411,\n",
       "        0.81466395, 0.94093686, 1.        , 1.        , 1.        ,\n",
       "        1.        , 1.        , 0.77189409, 0.77189409, 0.84928717,\n",
       "        0.97352342, 1.        , 1.        , 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'split2_test_score': array([0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 , 0.77235772,\n",
       "        0.68292683, 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.77235772, 0.74796748, 0.75609756,\n",
       "        0.66666667, 0.6504065 , 0.6504065 , 0.6504065 , 0.6504065 ,\n",
       "        0.76422764, 0.74796748, 0.76422764, 0.76422764, 0.7398374 ,\n",
       "        0.65853659, 0.6504065 , 0.6504065 , 0.6504065 , 0.74796748,\n",
       "        0.77235772, 0.75609756, 0.71544715, 0.73170732, 0.65853659,\n",
       "        0.6504065 , 0.6504065 , 0.6504065 , 0.77235772, 0.77235772,\n",
       "        0.77235772, 0.67479675, 0.73170732, 0.65853659, 0.6504065 ,\n",
       "        0.6504065 , 0.6504065 , 0.76422764, 0.75609756, 0.77235772,\n",
       "        0.70731707, 0.73170732, 0.65853659, 0.6504065 , 0.6504065 ,\n",
       "        0.6504065 ]),\n",
       " 'split2_train_score': array([0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.65376782, 0.65376782, 0.76782077,\n",
       "        0.73930754, 0.67820774, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.65376782, 0.65376782, 0.76985743, 0.76374745, 0.78207739,\n",
       "        0.85336049, 0.98778004, 0.65376782, 0.65376782, 0.65376782,\n",
       "        0.77189409, 0.76374745, 0.77596741, 0.82281059, 0.9898167 ,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.76374745,\n",
       "        0.77393075, 0.78207739, 0.86761711, 1.        , 1.        ,\n",
       "        1.        , 1.        , 1.        , 0.77189409, 0.77800407,\n",
       "        0.79429735, 0.92668024, 1.        , 1.        , 1.        ,\n",
       "        1.        , 1.        , 0.77800407, 0.78004073, 0.82281059,\n",
       "        0.9796334 , 1.        , 1.        , 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'split3_test_score': array([0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.77868852,\n",
       "        0.78688525, 0.68032787, 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.79508197, 0.82786885, 0.81967213,\n",
       "        0.72131148, 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.78688525, 0.82786885, 0.8442623 , 0.81967213, 0.70491803,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.82786885,\n",
       "        0.8442623 , 0.83606557, 0.80327869, 0.68852459, 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.8442623 , 0.83606557,\n",
       "        0.80327869, 0.68852459, 0.68852459, 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.8442623 , 0.82786885, 0.81967213,\n",
       "        0.67213115, 0.68852459, 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 ]),\n",
       " 'split3_train_score': array([0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.76219512,\n",
       "        0.71138211, 0.77845528, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.73373984, 0.7398374 , 0.7804878 ,\n",
       "        0.83536585, 0.99796748, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.72764228, 0.74593496, 0.7601626 , 0.81504065, 0.98577236,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.74593496,\n",
       "        0.75813008, 0.76219512, 0.8699187 , 1.        , 1.        ,\n",
       "        1.        , 1.        , 1.        , 0.75203252, 0.75813008,\n",
       "        0.7804878 , 0.92682927, 1.        , 1.        , 1.        ,\n",
       "        1.        , 1.        , 0.76422764, 0.75813008, 0.82926829,\n",
       "        0.97764228, 1.        , 1.        , 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'split4_test_score': array([0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 , 0.7704918 ,\n",
       "        0.66393443, 0.67213115, 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.7704918 , 0.7295082 , 0.72131148,\n",
       "        0.68032787, 0.6557377 , 0.6557377 , 0.6557377 , 0.6557377 ,\n",
       "        0.7704918 , 0.73770492, 0.7295082 , 0.71311475, 0.73770492,\n",
       "        0.64754098, 0.6557377 , 0.6557377 , 0.6557377 , 0.73770492,\n",
       "        0.71311475, 0.72131148, 0.72131148, 0.71311475, 0.64754098,\n",
       "        0.6557377 , 0.6557377 , 0.6557377 , 0.71311475, 0.71311475,\n",
       "        0.69672131, 0.64754098, 0.71311475, 0.64754098, 0.6557377 ,\n",
       "        0.6557377 , 0.6557377 , 0.72131148, 0.72131148, 0.71311475,\n",
       "        0.6557377 , 0.71311475, 0.64754098, 0.6557377 , 0.6557377 ,\n",
       "        0.6557377 ]),\n",
       " 'split4_train_score': array([0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.65243902, 0.65243902, 0.76626016,\n",
       "        0.73373984, 0.74593496, 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.65243902, 0.65243902, 0.76219512, 0.76422764, 0.79268293,\n",
       "        0.81707317, 1.        , 0.65243902, 0.65243902, 0.65243902,\n",
       "        0.76422764, 0.76626016, 0.78658537, 0.83130081, 0.9898374 ,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.76422764,\n",
       "        0.76422764, 0.80081301, 0.86585366, 1.        , 1.        ,\n",
       "        1.        , 1.        , 1.        , 0.76422764, 0.77235772,\n",
       "        0.80894309, 0.93089431, 1.        , 1.        , 1.        ,\n",
       "        1.        , 1.        , 0.76422764, 0.79471545, 0.82926829,\n",
       "        0.9796748 , 1.        , 1.        , 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'std_fit_time': array([2.41742878e-03, 1.90734863e-07, 4.10190833e-07, 4.00209441e-04,\n",
       "        1.46506228e-06, 3.99806145e-04, 4.90057117e-04, 9.81321731e-04,\n",
       "        4.00328846e-04, 1.50789149e-07, 5.76164530e-07, 4.15696997e-07,\n",
       "        1.12436544e-06, 4.10190833e-07, 1.90734863e-07, 8.97163759e-07,\n",
       "        4.91304519e-04, 1.80946979e-06, 9.53674316e-08, 1.30063836e-06,\n",
       "        3.16297988e-07, 2.43140197e-07, 1.78416128e-07, 1.46971083e-06,\n",
       "        4.00186902e-04, 4.88834947e-04, 4.90446491e-04, 9.53674316e-08,\n",
       "        4.00257139e-04, 1.50789149e-07, 4.89922634e-04, 1.90734863e-07,\n",
       "        1.58436277e-06, 4.90310237e-04, 4.00328704e-04, 4.00137912e-04,\n",
       "        1.16800773e-07, 4.00328675e-04, 1.16800773e-07, 4.00114074e-04,\n",
       "        7.48774263e-04, 4.90407519e-04, 4.00257309e-04, 2.34378924e-06,\n",
       "        7.48990934e-04, 3.87384339e-07, 4.90329667e-04, 4.00400318e-04,\n",
       "        1.12234137e-06, 4.00304865e-04, 4.90135007e-04, 1.72159325e-03,\n",
       "        8.95268811e-04, 8.94948929e-04, 4.00567236e-04, 4.90173941e-04,\n",
       "        4.00021236e-04, 1.60096885e-03, 3.99947802e-04, 5.63954175e-03,\n",
       "        8.95002166e-04, 8.94789101e-04, 1.20110522e-03, 4.90642071e-04,\n",
       "        6.32937475e-04, 1.02052489e-03, 2.37602562e-02, 1.35818191e-03,\n",
       "        1.41394249e-03, 1.96079302e-03, 4.49293870e-03, 1.49745931e-03,\n",
       "        4.00233393e-04, 4.08188207e-03, 2.82522168e-02, 4.93918352e-02,\n",
       "        4.90310237e-04, 4.90368632e-04, 9.80409451e-04, 6.32937493e-04,\n",
       "        7.47615057e-04]),\n",
       " 'std_score_time': array([4.90349959e-04, 4.90310284e-04, 4.91187881e-04, 4.90349125e-04,\n",
       "        4.90173987e-04, 4.90212913e-04, 1.78416128e-07, 4.90602326e-04,\n",
       "        4.90154518e-04, 4.90251785e-04, 4.90446491e-04, 4.90193437e-04,\n",
       "        4.90329691e-04, 4.90290788e-04, 4.00185766e-04, 4.00686420e-04,\n",
       "        4.00424043e-04, 4.90213215e-04, 4.90193390e-04, 4.90232393e-04,\n",
       "        4.89998792e-04, 4.00733976e-04, 4.90388074e-04, 4.00042545e-04,\n",
       "        4.00591053e-04, 4.00662461e-04, 4.00495685e-04, 4.90193437e-04,\n",
       "        4.00352506e-04, 4.90096135e-04, 4.00471812e-04, 4.90388120e-04,\n",
       "        1.16800773e-07, 4.90251831e-04, 4.90173964e-04, 4.90096089e-04,\n",
       "        4.00304809e-04, 4.89998722e-04, 1.78416128e-07, 4.00090313e-04,\n",
       "        4.00209498e-04, 4.90446468e-04, 4.00543270e-04, 7.49067400e-04,\n",
       "        4.90816328e-04, 4.90135007e-04, 5.09122765e-07, 3.87384339e-07,\n",
       "        4.89901475e-04, 5.13569337e-07, 4.00519382e-04, 4.00615203e-04,\n",
       "        4.00114216e-04, 4.00567292e-04, 4.90388259e-04, 4.89804116e-04,\n",
       "        4.00257139e-04, 4.90232324e-04, 5.35248383e-07, 4.90835835e-04,\n",
       "        2.33601546e-07, 4.00640028e-04, 4.00853196e-04, 4.00447874e-04,\n",
       "        4.00280963e-04, 4.90193437e-04, 4.90213006e-04, 8.03580262e-07,\n",
       "        4.90777440e-04, 4.00304922e-04, 1.85584914e-03, 7.50610262e-04,\n",
       "        4.00018735e-04, 4.89979520e-04, 4.89921520e-04, 3.99971037e-04,\n",
       "        3.23406696e-07, 4.90173941e-04, 3.23406696e-07, 3.16297988e-07,\n",
       "        4.90582772e-04]),\n",
       " 'std_test_score': array([0.00238217, 0.00238217, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.00238217, 0.00238217, 0.02356435,\n",
       "        0.04345715, 0.01184656, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.02655911, 0.03841229, 0.03224373,\n",
       "        0.01863698, 0.00238217, 0.00238217, 0.00238217, 0.00238217,\n",
       "        0.02454569, 0.03756986, 0.03940366, 0.04094287, 0.01278771,\n",
       "        0.00409062, 0.00238217, 0.00238217, 0.00238217, 0.0368263 ,\n",
       "        0.04622465, 0.03816753, 0.04035546, 0.01987411, 0.00409062,\n",
       "        0.00238217, 0.00238217, 0.00238217, 0.04625749, 0.04104206,\n",
       "        0.03624047, 0.01801507, 0.01787038, 0.00409062, 0.00238217,\n",
       "        0.00238217, 0.00238217, 0.04620948, 0.03605294, 0.04115794,\n",
       "        0.03046661, 0.01787038, 0.00409062, 0.00238217, 0.00238217,\n",
       "        0.00238217]),\n",
       " 'std_train_score': array([0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.0005945 , 0.0005945 , 0.0005945 , 0.0005945 , 0.00387711,\n",
       "        0.01044139, 0.0477565 , 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.0005945 , 0.0005945 , 0.01394136, 0.00929634, 0.00476533,\n",
       "        0.01583887, 0.00505539, 0.0005945 , 0.0005945 , 0.0005945 ,\n",
       "        0.01740342, 0.00729978, 0.00922623, 0.01145388, 0.00198846,\n",
       "        0.        , 0.        , 0.        , 0.        , 0.0071313 ,\n",
       "        0.00646818, 0.0126037 , 0.00495408, 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.        , 0.00865766, 0.00796623,\n",
       "        0.01263013, 0.00552857, 0.        , 0.        , 0.        ,\n",
       "        0.        , 0.        , 0.0056863 , 0.01272433, 0.00923604,\n",
       "        0.00278677, 0.        , 0.        , 0.        , 0.        ,\n",
       "        0.        ])}"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the complete results (list of named tuples)\n",
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXl4VEXWh9/qTjrp7BtJSEgCyI6RkIAMiyiColFwRdQZEFzQccFBRZ0ZR0BHxx1F/BxRNhkdQEXFAURQUcAFCARCAmFfQshK9q3T3fX9cZMmCVk6SXcSoN7nuU/3vbdu3dNZ+tw6der8hJQShUKhUChaiq69DVAoFArF+Y1yJAqFQqFoFcqRKBQKhaJVKEeiUCgUilahHIlCoVAoWoVyJAqFQqFoFcqRKBQKhaJVKEeiUCgUilahHIlCoVAoWoVLexvQFgQFBcmuXbu2txkKhUJxXpGQkJAjpezUVLuLwpF07dqVHTt2tLcZCoVCcV4hhDhuTzsV2lIoFApFq1CORKFQKBStQjkShUKhULSKi2KOpD4qKytJS0ujvLy8vU1ROBB3d3e6dOmCq6tre5uiUFw0XLSOJC0tDW9vb7p27YoQor3NUTgAKSW5ubmkpaXRrVu39jZHobhouGhDW+Xl5QQGBioncgEhhCAwMFCNMhWKNuaidSSAciIXIOp3qlC0PRe1I2kuEz/4lYkf/NreZigUCkWHQjkShUJxfrD4Bm1T2EVbPvgqR6JoMXq9npiYGNLT0wGIj48nPz+fsrIyYmJiMBgM5OTk2Nrn5uYSExNDTEwMoaGhhIeH2/ZNJlOz7r1o0SIyMjIc+nkUCkXLuGizthStx2g0kpiYaNtfu3at7X1iYiJ165sFBgba2s+ePRsvLy+eeuqpFt170aJFxMbGEhoa2qLrFQ0z9dupACy+bnE7W3J+UP3Uv+LBoe1sSfuhHAkw55tkUtILm2yXclprY89wsV+YD7PG9W+0zYsvvsgnn3xCREQEQUFBxMXF8dRTT/Hhhx+yYMECTCYTPXr0YNmyZXh4eDBlyhSMRiP79+/n+PHjLF68mKVLl/Lrr78yZMgQlixZAoCXlxePPPIIGzduxN/fn5dffpmnn36aEydO8PbbbzN+/HiOHTvGpEmTKCkpAWD+/PkMGzasXjtPnz7NxIkTKSwsxGw28/7773PFFVec0666pllQUFCTP5/6WLp0Ke+99x4mk4lhw4Yxf/58rFYrU6dOJTExESkl06ZNIyQkhMTERCZOnIjRaGTbtm0YDIYW3VOhULQeFdpqJ3bs2MEXX3zBrl27WLVqVa2ikrfeeivbt29n9+7d9O3bl4ULF9rO5eXl8cMPPzB37lzGjRvHjBkzSE5OJikpyfa0X1JSwlVXXUVCQgLe3t4899xzbNiwgS+//JLnn38egODgYDZs2MDOnTtZsWIF06dPb9DWTz/9lLFjx5KYmMju3buJiYlx+M9j7969fPnll/zyyy8kJiZiNptZvnw5CQkJ5OTkkJSUxN69e5k8eTITJ04kJiaGFStWkJiYqJyIQtHOqBEJNDlyqMaRQ9gtW7Zw0003YTQaARg3bpzt3N69e3nuuefIz8+nuLiYsWPH2s6NGzcOIQTR0dGEhIQQHR0NQP/+/Tl27JhtbuK6664DIDo6Gjc3N1xdXYmOjubYsWOAtrL/0UcfJTExEb1ez4EDBxq0dfDgwdx7771UVlZy8803O8WRbNy4ke3btzNo0CAAysrKiIiIYOzYsaSmpvL4448THx/Ptdde6/B7KxSK1qEcSTshpWzw3JQpU/jqq68YMGAAS5YsYdOmTbZzbm5uAOh0Otv76n2z2QyAq6urbT1FzXY128ydO5eQkBB2796N1WrF3d29QXtGjhzJzz//zJo1a5g0aRIzZ85k8uTJLfvgDSCl5N577+XFF18859yePXtYt24d8+bN44svvmDBggUOvbdC0Rqez51Z9W5Lu9rRnqjQVjsxYsQIvvnmG8rLyykuLmbNmjW2c0VFRXTu3JnKyko++eQTp9y/oKCAzp07o9PpWLZsGRaLpcG2x48fJzg4mAceeID77ruPnTt3OtyeMWPGsHLlSluWV25uLidOnCA7OxspJRMmTGDOnDm2e3t7e1NUVORwOxQdl6kik6kis73NUNSDU0ckQojrgHcAPfCRlPKVOucjgaWAX1WbZ6WUa6vO/RW4D7AA06WU6+3p83xh8ODBjB8/ngEDBhAVFcWgQYPw9fUFtEn4IUOGEBUVRXR0tFO+MB9++GFuu+02PvvsM0aNGoWnp2eDbTdt2sTrr7+Oq6srXl5efPzxxw63Jzo6mlmzZjFmzBisViuurq78+9//Rq/Xc9999yGlRAjBq6++CsDUqVO5//771WS7QtERkFI6ZUP7oj8MdAcMwG6gX502C4A/V73vBxyr8X434AZ0q+pHb0+f9W1xcXGyLikpKecca2uKioqklFKWlJTIuLg4mZCQ0M4WNQ9PT89Gz0dFRcns7Ow2suYsHeF3ez4zZd0UOWXdlPY24xymLI6TUxaf+7/c3ux9abjc+9Lw9jbjHO749y/yjn//0qo+gB3Sju97Z4a2LgcOSSmPSClNwHLgpjptJOBT9d4XSK96fxOwXEpZIaU8Chyq6s+ePs8bpk2bRkxMDLGxsdx2223Exsa2t0nNwsfHp9aCxGqqFyRWVlai06noqUJxoePM0FY4cLLGfhowpE6b2cB3QojHAE9gTI1rf6tzbXjV+6b6BEAIMQ2YBhAZGdl869uATz/9tL1NqEVSUhKTJk2qdczNzY3ff/+93vZ1HUg1dRcqKhSKCxtnOpL6yrDWTVW6C1gipXxTCDEUWCaEuLSRa+t7vK03/UlKuQAtdMagQYMaTpFS2IiOjlYOQKFQNBtnOpI0IKLGfhfOhq6quQ+4DkBK+asQwh0IauLapvpUKBSKi562TEt2ZgB7O9BTCNFNCGEA7gRW12lzAhgNIIToC7gD2VXt7hRCuAkhugE9gW129qlQKBSKNsRpIxIppVkI8SiwHi3bapGUMlkI8QJaJsBq4EngQyHEDLQQ1ZSqTIFkIcRKIAUwA49IKS0A9fXprM9wDtUlrKeuabydQqFQXEQ4dR2J1NaErK1z7Pka71OA4Q1c+xLwkj19KhQKhaL9ULmZihaj9Ehax/FJkzk+ybGlZhzBne8mc+e7bTfQV5z/qFpbihaj9EgUCpgTVAzAyna2oy5taZdyJADrnoWMpKbbZezRXu2R+wyNhusbr96i9Ehqo/RIFIrzExXaaieUHkltLko9kowk+x5gFACUmMyUmMztbYaiHtSIBJocOdhwYNaW0iOpzcWoR7IfbV4oqp3tOAdTSXtboDjPUCOSdkI2oUcyf/58kpKSmDVrFuXl5bZzztAj2bFjR6OT3dV6JOHh4UyaNMkp1X9llR5JYmIiiYmJpKam8o9//IPAwED27NnDiBEjmDdvHg8++KDD761QKFqHciTthNIjqY3SI1Eozl9UaKudUHoktVF6JArF+YtoLMRyoTBo0CBZczIbYN++ffTt27edLNIoLi7Gy8uL0tJSRo4cyYIFC86rUvJeXl4UFxc3eL61WVwtpSP8bu1h/TX9ARi7oWOt2eiodt2xQJubWzmtYxUWvZDtEkIkSCkHNdVOhbbaEaVHouiIWKTEchE8YCochwpttSNKj6RtmPrtVAAWX7e4nS1RKC5MlCNR2FB6JAqFoiWouINCoVAoWoVyJAqFQqFoFcqRNIOp3061xdsVCoVCoaEciaLFqDLyCoUC1GS7ohWoMvIKhQLUiKRdefHFF+nTpw/XXHMNd911F2+88QYAH374IYMHD2bAgAHcdtttlJaWAloNrj//+c+MGjWK7t2789NPP3HvvffSt29fpkyZYuvXy8uLZ555hri4OMaMGcO2bdu46qqr6N69O6tXaxL3x44d44orriA2NpbY2Fh++eWXBu08ffo0I0eOJCYmhksvvZTNmzfX265r1661RiDNZenSpVx++eXExMTw8MMPY7VaMZvNTJo0iejoaC699FLmzZtnq/pbXQW4uaMZhULhWJw6IhFCXAe8g6av/pGU8pU65+cCo6p2PYBgKaWfEGIUMLdG0z7AnVLKr4QQS4ArgYKqc1OklK3KWX1126vsP7O/yXbVbeyZJ+kT0IdnLn+mwfM1y8ibzWZiY2OJi4sDtDLyDzzwAADPPfccCxcu5LHHHgPOlpFfvXo148aNY+vWrXz00UcMHjyYxMREYmJibGXkX331VW655RZbGfmUlBTuuecexo8fbysj7+7uzsGDB7nrrruou/q/muoy8n//+9+xWCw2x+ZIapaRd3FxYdq0aSxfvpxLLrnEVkYeID8/Hz8/P959913mz5/vlErECoWieTjNkQgh9MB7wDVAGrBdCLG6SqcdACnljBrtHwMGVh3/EYipOh4AHAK+q9H9TCnl586yvS1QZeRrczGWkVcoLhScOSK5HDgkpTwCIIRYDtwEpDTQ/i5gVj3HbwfWSSkd/xhcRWMjh5o4coV0U2Xkv/rqKwYMGMCSJUvYtGmT7ZwzyshbrVbc3d0btKe6jPyaNWuYNGkSM2fOZPJkx2qNV5eRf/HFF885t2fPHtatW8e8efP44osvWLBggUPvrVAoWocz50jCgZM19tOqjp2DECIK6Ab8UM/pO4H/1jn2khBijxBirhDCrZ5rOjyqjHxtVBl5heL8xZkjElHPsYYew+8EPpdS1vo2E0J0BqKB9TUO/xXIAAzAAuAZ4IVzbi7ENGAaQGRkZHNtdzqqjHxtVBl5RVNMWVmVVDGtfe1Q1IOU0ikbMBRYX2P/r8BfG2i7CxhWz/HHgQWN3OMq4H9N2RIXFyfrkpKScs6xppiyboqcsm5Ks69riKKiIimllCUlJTIuLk4mJCQ4rO+2wNPTs9HzUVFRMjs7u42sOUvd362jf2+O4tsx/eS3Y/q1txnnsGZ0X7lmdN/2NuMcOqpdEz4YICd8MKC9zTgHR9gF7JB2fN87c0SyHegphOgGnEIbddxdt5EQojfgD/xaTx93oTmgmu07SylPC20S4GZgr6MNbwhHV4+dNm0aKSkplJeXc88995y3ZeTXrl1LWFiY7XhZWRlDhw5VZeQViosEpzkSKaVZCPEoWlhKDyySUiYLIV5A83Krq5reBSyv8n42hBBdgQjgpzpdfyKE6IQWOksEHnLWZ3A2qoy8QqG4EHDqOhIp5VpgbZ1jz9fZn93AtceoZ3JeSnm14yxU1ESVkVcoFC1BxR0UCoVC0SpUrS2FQmEj9Uwq/x0hOBwi+Gj1rXQydiLIGESgMZAg9yCCjFWbh/bq7eptW7OkuHhRjkShULAzcycfJX3E5lObMXQR9D8p8evdhdyyXI4VHCO7LJtKa+U51xl0BptzCTQG1nY8xqBa59z05+WSL4UdKEfSDI5P0lZzRy1z/DoKhaKtkVKy+dRmPkr6iF1Zu/B38+fRmEcJ/Os7eJggfs68Wm0LTYXkluWSU5ZzdivPIadUe59WnMbu7N2cKT9T7/18DD61HEtNR1Nz83PzQydU1P18QjkSRYvR6/VER0fb0n/j4+P59NNPcXNzY+jQoaSkpJCenk5QUBCgrVYfPXo0ABkZGej1ejp16gTQ7EWFixYtIj4+/rwuI++it+LtaQKzCVzabkGl2Wrmu2PfsXDvQg7kHSDUM5RnL3+WW3veitHFyFrTO+dcI4TA180XXzdfuvt1b7T/SmsleeV5ZJdln+t4qra9OXvJKcuhzFx2zvV6oSfQPVAb4XhUjXDcA8nqLfAtkQwpyyXQGOiwn4ei9ShHomgxSo+kdfSMKiCycwm83gP6xEO/m+CSq8HFOSGgCksFXx/6msV7F5NWnEY33278c/g/ie8ej6vO1WH3cdW5EuwRTLBHcJNtSytLbc4luyybnLKcWs4nuzSbfbn7OFN+Bsvl2ijl45VX0dWnKwODBxIbEktscCwR3hFqrqYdUY6kHXnxxRf55JNPiIiIICgoiLi4OJ566ik+/PBDFixYgMlkokePHixbtgwPDw+mTJmC0Whk//79HD9+nMWLF7N06VJ+/fVXhgwZwpIlSwBNj+SRRx5h48aN+Pv78/LLL/P0009z4sQJ3n77bcaPH8+xY8eYNGkSJSUlAMyfP59hw4bVa+fp06eZOHEihYWFmM1m3n//fa644opz2nXt2pUdO3bYRiDNZenSpbz33nuYTCaGDRvG/PnzsVqtTJ06lcTERKSUTJs2jZCQEJseyXlbIsVqJTigjLxCA/4jb4T9/4Pd/wU3H+h1HfS/GS4ZDa4NF9O0l2JTMSsPrGRZyjJyynK4NPBSnhr0FKMiR7V7CMnD1YNI10gifRovY2QpyWbLny+nONBK5hV/YFfFGb4//D++PPQlAEE6NwYaAok1BBBrCKSXiw8ubfTZ/lRwBimATa+CwbNq8zr73s2r9r7BC/Qtc9xSSirMVkpNFkoqzNqryUxJhZmSCgulJjMlJgulFWbSzozGKg2cKTER4Onc/w/lSICMl1+mYl/TeiTl+7U21XMljeHWtw+hf/tbg+eVHkltLjo9ktOJuLtZOXjcE/+b/w/Mb8PRnyHlS9i/BpJWal841U6lxxhwNTbrFmfKz/CflP+wPHU5RaYi/tD5D/zrin8xJHRIx396t1ogfRcc2ggHN6A/lcCV3SUWK+j3/aw1AY646Nnp5spOQzm73ErZ4KItkvWwWokxmRlYUUmsyUy0qRJjwwW3W8W15jJ0Etj0st3XWHWuVOo9qNR7YNIZqdAZKRfulGGkBHdKpDtF0o0iqxuFVgMFZjfyLa7kVRookm6USndKcKcUd4ql9mrChdolDq9GJyqUI7mQUXoktXGmHsmAz/awo6uFld1WMiJ8BGFeYU1f5GxS11KY5oZ5vzs5HyzAIy4W9+gr0PUcAzdWO5WvYd83sPdzcPWEXmOrnMo1YPBosOv04nSWJi9l1cFVVFgqGB05mvui7+PSoEvtMu2wZxQbQkexZ91++oX50D/Mh66Bnuh1TnY+xVlw+Ac4uEF7LTsDCAiPgyufYes7CykodiV+4z5AWwTXo2q7o6qLjJIMdmbuZGeWtv1f3iEkEhfhQr/AfsSGxDIweCADgwfi7+7fYlNLKszszygi5XQh875fTHllEF39L0WaSpAVxYjKUkRlMR6yAg9RjiflZ1+pwFOU48HZ416iHG9RjJculzC0c0bKcZfltW/cyEBGChesBk9w9US4eXKk8ATlQtDDdRvg1eLPag/KkUCjI4eaODJrq05FmFooPZLatFaP5Pt+Zo4HWNn9m9Z3N99ujAgfwYiwEcSFxrVLWmrZz6s59UsABinInquJgQpXV9yjo/GIi8MYF4vHVf9Ef8NbcHwLJH+lOZXkVeDqAT2v1eZUeo3VwiXA4fzDLNq7iLVHtLmqG7rfwL3R99Ldt/HJ8bpsDL2SFN/e7NtyhEqL9ndqdNXTp7M3/cN86NfZl35hPvQJ9cbdVd/yH4LFDKd2aI7j0EY4XTXf5tlJ+3w9r4Huo8BTm1gveGlZk12GeoYS3z2e+O7x2jUVBezO3s3OzJ3sytrFJ/s+YUnyEgC6+3ZnYPBA4kLiGBg8kHCv8HpHatlFFaScLiQ5vYCU9EJSThdyNKeE6n9hva4/HoZMQvw88TD44ummx8PggqdBj6ebCx5u2nsPg8vZc256PA0ueFS1cXPR1T9KtFqhshRMJWAqrnotqbNfDKZihKkEfY3jhcVpuEsJOud/zStH0k6MGDGCBx98kL/+9a+YzWbWrFljC2fV1SMJD69XxqVVFBQU0KVLF3Q6HUuXLm1SjyQ8PJwHHniAkpISdu7c6XBHMmbMGG6//XYef/xxgoKCyM3NpaSkBKPRiLu7OxMmTKBbt2489JBWWs1ePZKcshyOB1q5JtmFR577gq2ntrLl1BZW7F/BspRluOvdGRw6WHMs4SOajNU7AvPhRNJW54GLjvQgHaM+30LZrl2U7kigdGcCuYsXw4cfghC49eypOZW4a/C4+ylcK4+cdSopX4GLkaRLhvGRUc8PBfsxuhi5s8+d3NP/HkI9m5+IUGaykOLTm2E52/howYsczCqyfXkmpxfy9a50/vPbCQB0Ai7p5GUbtVQ7mEbDKIWnNadxaCMc+RHKC0DooMvlcPVzWggvdAA4qNinr5svI7uMZGSXkYCWcJCck6yNWDJ38t2x7/ji4BcAdDIG08fvMvz0vbCWduN0jh8p6cVkF1XY+osIMNKvsw83DQjXPnOYD4+vGIoQ8NE9jzrE5lrodNoci5sXENKsS19ZoEUOVvo6/vujLsqRtBNKj6Q2ztIj+eHED0gB0af0dPftTnff7kzqN4kycxnbM7bbHMvmU5sBiPSOZHj4cEaEj2Bw6GCMLs2bl2gKaTKRNuNJLBU68sLAisAlIADv0aPxrkqNtpaVUbZ7D6U7EyjbkUDh16vJ/+9yAFzDwjDGxWGMncNBz+N8WLSO300H8S6z8mBxGX/s1A9/9+6gbzj01Rg/H8ymUm8gJm8vBhcd/cN86R/me9Z+KUnLK6v1dL7t6Bm+TjxbwLOzrzv9OmtfspeGGomRBwjO2ow4tBEyq4p1e3eGvuM0x9H9KjC2PMzUHNz0bsSGxNIvYACX+99Osnc+v51KZm/ubjKK9pFZuA2d60YAhHAnILIXcX6XMazLYK7reTnBXueGiDr6dFNbIBoLsVwoDBo0SNadSN63bx99+/ZtVj+OXpBYXFyMl5cXpaWljBw5kgULFpxXpeS9vLwoLi5u8Hxrs7haSs3f7UMbHmJ/6i88u86N69Zsb/CaE4Un2HJqC1vTt7Lt9DbKLeUYdAYGhQ5ieNhwRnQZQTefbq2apJZSkjFrNvkrVxI+1sivKdqT7tgNyY1fZzZTnppKWUICJQkJFGz7FX2e9nBRbBRU9u9Ot+hofH1yMBZtRpSeBr0b9Bithb96Xw/uvo3eo5onViaybttBXts9h3Eb7FdoOFNiYl9V+Cf9+EH80n+ib/E2humS8RZlVEo9qW79OR00HGuPMXTpPYieIT4YXOwfeawd0w+A+I0NqXXXT36piZR0bUSVcrqQlPRCDmUXY7Fq331ebi42x9e3szfB/mXky1SSchLZmbmTwwWHAS2tuX9gfwaGDCQuOI6Y4Bh83Xy5o/rJf1rHKnjqCLuEEAlSykFNtVMjkmbg6BXtSo/EuRSZivg943eGn9Ij6hXsPEukTyR3+9zN3X3vpsJSQUJmguZYTm3l9R2v8/qO1wnzDGNE+AiGhw9nSOcheLo2PIqrj/zly8lfuZLAfiX4jJsIKZ/ZdZ1wccGlTy9+MhxgkfcRjg0sZWBlBJPNg+hzUlKxM5H8xV+RDwg3d4y9R2EMkXicScS4dx16d1dtfUq/m6B3PBj96r1PpcXK9/uyiC7Yh15a7f9g5goCMn5h+OGNDD+0EbK17Ebp34XcsFvZ5TGYnyr7sivTwr6TRZQdqYDvtuKqF/QIrp530b7I+4X54OPe8tRYbbR01mGkpBeQXnB2wjrUx53+YT5c2z+Efp196B/mSxd/I7pzEgl6c0vP8QDkl+eTmJ1om8RflrKMxXs1baIefj3IcbFitEJSdhKRPpH4utnntC8klCNpR5QeiXP5Oe1nzFYz0aeal/ropndjWNgwhoUNg8FwqvgUW09tZeuprfzvyP9YeWAlLjoXYoNjbWGwnn49Gx2tlGzbRsZLL+MV24tOPTZB7xuAph1JaWUpXxz8gqXJS8kszaS3f29ev/INrom6Br3u7ES3OTeX0oQEyhJ2UpqQQO4P+8i1ALow3Dr74LE7CY+NmzCG/AXX/iO17K/e8eARYOtj29EzFJRVMiDPjpHImaNn5zqO/qxNCOsNEDUcBk6CntcggnoRJAQjgZFVl1mskmO5JbXmXTalZvF5Qpqt6+p5iP5hvjYH09m3djJIpcXKwcxim8NITi8g5XQhReVaMolOQPdOXgzuFmDrq29nbwK9mp9Y4efux1URV3FVxFUAlJnL2Juz1zaBfyTvEPl6uHutptvnY/Ah0ltbGxPpE0mkdyQR3hFE+UTh5+bX8VOvW4AKbSkuOKp/t09seoJdWbt4emkxOgRj12xrdd+VlkoSsxPZfGozW09t5UCeljYd7BGsjVbChvOHsD/gY/CxXWNKO8WxCRPQ+/vT9S5/9Bm/w5OprB+rpW7XF9oqqCjg0/2f8um+T8mvyCcuJI77o+9neNhwu76IrCUllO3ZY5vAL0vcjSzTypG4+oBHQCnGYDMeA2MwXDEB0Xccz288zYrtJ3ll27O4WStrh5Aqy+DYFtu6Ds5o4R78u2rpyD2vga4jbBlkzSWrqFwbSdQYTRzLPZsZ5e/hSnBGCr6VhZREj+JgZjEmizZqqs4oq3Y6/cN86R3ijdHQioyyZjBhwQBMAv4y9m1OFJ3gZNFJThSe4ETRCU6XnMZaY3Tn7epNhE/EWUdT4zXAPcChTkaFthSKVlJhqWDLqS3c2P1GdPzPYf266l0ZHDqYwaGDeSLuCTJLMvkl/Rc2n9rMhmMbWHVwFXqhZ0CnAZpjCYjD/ZE5SIuFiHfnol9xNVx6S4NZSVmlWXyc/DGfHfiMUnMpV3a5kvuj7ycmuHlrd3SenngOHYrn0KEAyMpKyvfv10YtOxIo3rGNgmOFsO0I+iUvY+w0h1s6BTKgx1C8RTkmdJBz8KzjOL4VzOXg4g5dr4AhD2oT5YGXtPpnChDs7U5wb3dG9T5bVqW4wkxqRqFtfuOX7CNkunci2tPA1BFdbSOWbkFtsMalEQQCNwmjIkedc85kMXGq+JTNsZwo1BzN3py9fHf8u1pOxtPVs9boJcI7wuZkgoxBHXokoxyJ4oLkt/TfKDOXMTpyNMUOdCR1CfEM4Zaet3BLz1swW83syd7DllNb2HJqC/N2voP+KytDDkh+fHw4l2R9zVBzCX69bzinn+OFx1m8dzGrD6/GKq1c1+067r30Xnr593KIncLVFWN0NMboaJgyBSklpqPHKEtIoHTrRoq27cB9Vwn9dm1E6ME9qJyyF4ZiDKiEwJ4w6F5t8j5qeLNX2LcULzcX4qICiIvSwm9r378TgPiSJrs0AAAgAElEQVS3mjfZ3p4Y9Aa6+Xajm2+3c85VWipJL0mv5WROFJ0gNS+VH078gFmabW2NLkbb6KWWo/GOpJNHp3YvdaMcSTP48s2dANzy5Pk1KX4x8v2J7/Fy9eLy0Mv5oY3u6aJz0YoIhsQyPXY6x995g9L9C9l2ez+W++2n4MDv6CLDufTgUkaUHcUlEHQSnvrpKTYc34CLcOHWnrdyT/97iPCOcKqtQgjcunfDrXs3/Cbczmvf7mfl+l2sGZhP/qI3KM81cGxDJ3zjr6HTn5/DNaTpAoyK5uGqdyXKJ4oon6hzzpmtZk4Xn9YcTA0nczDvID+e/BGz9ayTcde7nw2XeUcS4RNBlHcUlcg2+4J36n2EENcB7wB64CMp5St1zs8FqseDHkCwlNKv6pwFSKo6d0JKOb7qeDdgORAA7AQmSSlNzvwcivMLKSWbTm7iii5X4NrC4nitpej77yl9fyE+48cx+cVX+ZPVQvJ7l7EloDNbhY73d7+PvF57ivQ8tYUp/acwqd8kgoxtmypdzfrkDHr160bI5D+Q8PG7iE6S2OvvJe/jZRT+uJXA++8jcOpUdMa2GY1c7LjoXIjwiSDCJ4LhDK91zmK1kFGawfHC45wsPGlzNkcLjvJz2s9nBcjcQEg4mHeQnv49nWuvszoWQuiB94BrgDRguxBitZTSNi6VUs6o0f4xYGCNLsqklPUFhl8F5koplwsh/g3cB7zvjM+gaJyOqkdisprIq8hjdORoB3zK5lN+4ADpM5/GPTqazi+8gBACfeZeLjtzisuu+DsPD/wjeeV5fPDYCEoN8NTc72pNzrc1h7KKOZxdwuShXW3HpE4QMnMm/nfeSdbrb5Az713yV35G8JNP4HPDDYh2TOu+2NHr9IR7hRPuFQ51ysZZrBaySrM4UXSC59Y8gElAZ8/OTrfJmX8NlwOHpJRHqkYMy4GbGml/F/DfxjoU2mzT1cDnVYeWAjc7wFZFC6hO861eQ7J27Vr8/PzOOV5NtR5JYmIiDz30EDNmzLDtN7cM/KJFi8jIyKj3XLlZW0x4Rfi5pe6djSU/n7RHHkV4etBl/rvoqmuYpa7VSoH00gpw+rv7M/AYDD9AuzoR0EYjANf2P7cEhyEigi7z3iFq2ce4BAaSPvNpjt15F6U7d7W1mQo70Ov0dPbqzJDOQ/C36gix6PAyOLdgIzg3tBUOnKyxnwYMqa+hECIK6Aa1wtnuQogdgBl4RUr5FRAI5Etpm4VKq7pPfX1OA6YBREY2Xj9p88oD5JxseIV2NTlp2mri6rmSxgiK8OKKOxqfKFV6JLVxhB6JlJJySzlDw4bi4dqyMiEtRZrNnHriCcwZGUQt+xjXkBpfzKlrIWIIeLZP6KoxvkvOYEAXXzr7Nhy28hg8mK6fraTg69Vkz53L8bvvxif+eoKffBJXJ9SCU5xfOHNEUl+uWkOLVu4EPpdS1qwcGFmVv3w38LYQ4pLm9CmlXCClHCSlHFQdPulI1NQjWbVqVS0tkFtvvZXt27eze/du+vbty8KFC23nqvVI5s6dy7hx45gxYwbJyckkJSXZFgFW65EkJCTg7e1t0yP58ssvef755wFseiQ7d+5kxYoVTJ8+vUFbq/VIEhMT2b17t1PKyNfUI0lMTMRsNrN8+XISEhJseiR79+5l8uTJTJw4kZiYGFasWHHOaKbcUo7FammXsFbma69R8suvhM6Zg7Hmzyj/BGQkaaVKOhinC8rYnVbAtf2bLvAodDr8brmZS75dR9DDD1P0w48cvj6erLfmYikuaQNrFR0VZ45I0oCaqSddgPqXQmuO5JGaB6SU6VWvR4QQm9DmT74A/IQQLlWjksb6tJumRg7VODJrS+mR1MZReiSFpkIAroy40uE2Nkb+F6vI+3gZAfdMxu/WW2qfTF2nvdaT9tvefJecCcBYOxxJNToPDzpNfwy/OyaQ9dZb5C5YQP6qVXR6fDp+t96K0LfNQkBFx8GZI5LtQE8hRDchhAHNWayu20gI0RvwB36tccxfCOFW9T4IGA6kSG0Z/o/A7VVN7wG+duJncBpN6ZHMnz+fpKQkZs2aRXn52VpBztAj2bFjByZTw4lv1Xok4eHhTJo0ySnVf6v1SKrnTFJTU/nHP/5BYGAge/bsYcSIEcybN48HH3yw0X6KTEUY9AYC3AMabedISnftImP2bDyHDSV45sxzG6SuhaBeENSjzWyyl/XJGVzSyZMewc2Po7uGhhL+2mt0XbkCQ2QkGf94nqO33U7Jb785wVJFR8ZpjqRqxPAosB7YB6yUUiYLIV4QQoyv0fQuYLms/c3aF9ghhNiN5jheqZHt9QzwhBDiENqcyULOQ0aMGME333xDeXk5xcXFrFmzxnaurh6JMygoKKBz587odDqWLVvWpB5JcHAwDzzwAPfddx87dzY9R9RcxowZw8qVK8nJyQG0DK8TJ06QnZ2NlJIJEyYwZ84c273r0yOpsFRQYa7A3aX1Ouf2UpmZSdr06biEhhL+1lsIlzqD/LJ8rbRIBwxr5ZWY+P3omWaNRurDeNllRH3yH8LnvoW1qIgTU6Zy8uFHqDh61EGWKjo6Tl1HIqVcC6ytc+z5Ovuz67nuFyC6gT6PoGWEndcoPZLaOEKPpMik/ZyM+rZZ62AtLyftkUeRJaVELFqE3q+eqrqHNoLV3CHDWt/vz8Jila12JKAtcPS5/nq8rr6aM0s/JveDDzgybjwBf7yboIcfRu978VXEvaiQUl7wW1xcnKxLSkrKOcfamqKiIimllCUlJTIuLk4mJCS0s0XNw9PTs9HzUVFRMjs7u42skfJw/mF5KO/QOb/bb+MHy2/jBzv0XlarVabNnClTeveRhRs3Ntzws6lSvtpdSov5nFPfjuknvx3Tz6F2NYf7l26Xf3h5o7RarbWOrxndV64Z3bdVfVdmZ8v0fzwvU/r2k6mXD5G5Hy+TVpOpVX06wi5nMOGDAXLCBwPa24xzcIRdwA5px3esWlXUjkybNo2YmBhiY2O57bbbzls9krrl5MvKyoiJiWlTPZJKSyVllWVttibjzKLFFK7+hk6PT7cpG56D2aQVPOx9Heg61gR0qcnMzweyubZfiFOKAboEBdH5hTl0+3IV7v37kfnSSxy56WaKNm1qdH5QcX6iam21I0qPxHFUh7W8Dd7kkOPUexVv3kzWm2/iPXYsgVUa8vVyfCtUFHbIsNbPB7KpMFsdEtZqDPfevYlYuJDiTZvIevU10h76M57DhhH87DO493JMQUpF+6McicJGdHR0hxCkagmFpkIMegNu+uYLFzWHiqNHOfXEk7j17EnYv15u/Gk+dS24GDVN8g7G+uRM/Dxcubyb87PbhBB4jxqF14gR5P13OdnvvcfRm2/Bb8IEOk1/DJfAQKfboHAuKrSlOO8xW82UVpbibfB2qmaDpahIK3/i4kKX995D59HIynkpYf9auGQUGNp2hX1TaJK6mYzuE4KLvu2+AoSrKwGTJ9Fj/bf4/+mP5H/xBYfHXkfuRx9hbST9vJrT4Y9zOvzxNrBU0VyUI1Gc9xSbipFIp86PSIuF9KdmYjpxgvB33sbQpYmyIBlJUJjWIdN+fzuSS2G5mbH11NYC539h6/38CP3b3+i+ejUegweT9cabHIm/gcJv16v5k/MU5UiawYo5z7JizrPtbYaiDkWmIlx0LhhdnJf2m/32OxT/9BOhf/8bnpfbkX2euhYQ0Os6p9nUUtYnZ2B01TOyV/uWDnLr3o2I9/+PyEUL0Xl4cOovf+H4pEmUJdmhGa/oUNjlSIQQXwghbhCinWW4FIo6WKWV4sriRsNafQL60CegT4vvUfC/NeR++CF+d9yB35132ndR6lqIuBy8GhGEMni2WOO8pVitku+SM7myVyfcXTtGJpnnsGF0+3IVoS/MwXT0GMcmTCD9mWepzMxsb9MUdmKvY3gfrXjiQSHEK0KIlv9XKi4Y9Hp9rfTf+Ph48vPzbem/BoPBtlIdtNXqMTExxMTEEBoaSnh4uG2/sRIt9VFdRr7YVIxVWp0W1ipLTub0c89hHBRH6HN/t28OpiANTu/ukGGtxLR8sooq6i0Z354IvR7/O+7gkvXfEvjAAxSuW8fh664ne/57WEtL29u8Rhlw4GEGHHi4vc04h7a0y66sLSnlRmCjEMIXraTJBiHESeBD4D9Sykon2qjooNRN81279mwRg8TERLp27VqrfbUeCcDs2bPx8vLiqaeeatG9Fy1aRGxsLJ08Nb1qZ5SMN+fkkPbIo+j9/enyzjsIezVTOnCRxvXJGbjoBKP7dCxHUo3ey4vgJ5/Ab+IdZL35Jjnz55P/2WcEPzFDS2BwYjKFouXYnf4rhAgE/gRMAnYBnwAj0AonXuUM49qKH5csIOv4kSbbZR3T2tgzTxIc1Z1RU6Y12kbpkdSmJXokOoOONZvWoHNw1FWaTKRNfxxLfj5dP/2keSmqqWshsAd06ljrJKTUwlp/6B6Ir0f7SBDbi6FLF7rMnUvppElk/usV0p95lss8Qzga1bZVnRX2Ye8cySpgM5qu+jgp5Xgp5Qop5WOA8+W3LkCUHkltWqJHsuQ/S/j8x88J8nKsWJSUkowX/0nZzp2EvfwS7v362X9xeQEc3dwhw1qHsoo5mlPSYLZWR8QjNpauK5YT9tqruJpKuDTlM8r2Jre3WYo62DsimS+l/KG+E1ITnzqvaWrkUE31SGTirFdafU+lR1KbluiRFFcW4yt88XR17IR13qefkv/ZZwROm4ZPfHzzLj60EayVHTasBXBNP+euZnc0QqfDd/x4ElccZeCej8mYNYuuK1co3ZMOhL3xgL5CCFtp0yq9kI43u3Qe0Vi+vNIjsU+PpLSyFC9XL/QOrGNV8vs2Ml/+F15XXUWnv7RgLUXqOvAI1DK2OhjrkzOJifAj1Lftyuw7EouLO0ejrqQ8OZm8TzpWeaGLHXsdyQNSyvzqHSllHvCAc0y6OFB6JLVprh6Jh5cHBYUFDs3WMqWlcerxxzF07UrYG68jmltw0lIJB7/T1o50sCKNp/LLSDpV4PTaWs4mN6AXniNGkP3221RmZLS3OYoq7A1t6YQQoqqsMEIIPWBnCouiPpQeSW2aq0dy+923M2vGLN70epPt27bX0m1vCdaSEk1bxGol4r356L1aMPV3/BdtjqR3M8NhbcB3VWGt82l+pF6EIHTW8xy5cRyZL71Ml3fntbdFCrBPjwR4HfgMGA1cDawE3rTn2o6wKT0S59CeeiQHzxyUR/OP1nuu7u/22J8myWN/mtRgX1aLRZ58bLpM6dtPFm3e0nKj1j4t5YvBUlYU29XcGTopDTHxg1/kmDc32dX2o8n/lh9N/reTLWo+Ne3K/mCBpgXz/Q/tbJWU/3zyA/nPJz9obzPOwRF24WA9kmeAH4A/A48A3wNPO9qpXWwoPZKWUWGuoMJSgbfB2yH95bz/PkXffUfwzJl4jRjesk6qizR2v8ru1erLH+vP8sf6t+x+zeBMiYltzZDU9UBgpGOv1wicOgW3nj3I+OeLWKtS2BXth70LEq1oq9vfb07nQojrgHcAPfCRlPKVOufnAqOqdj2AYCmlnxAipupePoAFeElKuaLqmiXAlUBB1XVTpJTnZe1zpUfSMgpNhQAOmR8p2riRnHfn43vTeAKm3NPyjjKToeAEjHyy1TY5mo37MrFK7HYkpS4BlLv48d8XficowotOEd4EdfEiKMIbd8+Osf5EuLoSOmcOx+/+I9nz3yPkGfVc257Y5UiEED2BfwH9AFvKh5SyeyPX6IH3gGuANGC7EGK1lDKlxvUzarR/DBhYtVsKTJZSHhRChAEJQoj18uyE/0wp5ef22K6wn/NFj6TIVIS7izuu+tZ9qZUfOED608/gHh1N6AsvtK4Eva1IY8dbP/JdcgbhfkYuDW/a8aZsSafcNQBXSwnegZ04tT+PA7+frXnlFeBGUBfvWg7GO9DdqeX7q3G31t73iI3Fb8IEznz8Mb7jx+Het6/TbVDUj72T7YuBWUD1CGIqNDn2vRw4JKU8AiCEWA7cBKQ00P6uqnsgpbQtapBSpgshsoBOQH4D1youEiotlZSZywj2aKQYoh2Y8/JIe/gRdJ6edJn/Ljq3Vgpipa6FLoPAu2NNZpdUmPn5YA53Xx7Z5Jd92v4z/PRpKq6WErxMp7nxEW1tU2mhidy0YrJPFpGTVkzOySKOJ+VQncHu5uGijViqHExQhDf+nT3Qt4HWSfCTT1D0ww+cnjWbrv/9VK0taSfsdSRGKeX3VZlbx4HZQojNVH3xN0A4cLLGfhowpL6GQogooBvaPEzdc5ejZYgdrnH4JSHE82hzNc9KKSvs/ByK8xxHhLWk2cypGU9gzswkatnHuIa08su/MB3Sd8Ho51vXjxP46UA2JjskdfMySvh2wV58QzyQRw7Xekr08DHg0S+AiH5n1RQrTRZyTxWTc1JzLDlpxSRvPoW5Uhs26FwEAZ09tVFLRJWT6eKFwehYUVa9nx8hzz5D+synyVuxgoC773Zo/+cz5cJ5sgp1sfe3Wl5VQv6gEOJR4BTQ1CNhfY8/Da3CuxP4XEpZazGDEKIzsAy4p2qeBuCvQAaac1mAlgjwwjk3F2IaMA0gMjKyCVPtI+uDPQAEP3iZQ/pTNJ8iU5EmqevS8hFE5quvUfrbb3T+178wOmKVfmpVscoOuprd38OVwV39G2xTXlzJ/97bg04vuPGRy1jz5O4m+3U16Ant5ktoN1/bMatVUpBVqo1cThaTk1bMsaQc9v1y2tbGp5ORTlXzLdUOxtPP0KrQmM+NN1Lw5ZdkvzUX79FjcA1p3WhV0XzsdSR/QZsMnw68iBbeampmMg2IqLHfBah/dlZzJI/UPCCE8AHWAM9JKX+rPi6lrP6rrBBCLAbqLR8rpVyA5mgYNGiQkl27ADBbzZRUlhBkbHltrfwvviBv2TIC7pmM3y03O8aw1HUQ0B069XZMfw7CZLbyw/4srusf2qCkrqXSyroPkijJq+DmJwbiE9Typ1idTuAf6ol/qCe9BmvHpJSUFphqOBft9fCubNt1Rm/Xs6GxSO3VL8QDnc4+5yKEIHTWLI6MG0/mK/+iy9y5Lf4MipbRpCOpmjS/Q0o5EyhGmx+xh+1ATyFEN7QRzJ1omiZ1++8N+AO/1jhmAL4EPpZSflanfWcp5WmhPcLcDCg5tXZCr9cTHR3N2rVrCQsLIz4+nk8//RQ3NzeGDh1KSkoK6enptmrAubm5jB49GoCMjAz0ej2dOmkqfdu2bWtyUWGxqRgAb4M3ixYtIj4+ntBQ+1dql+7cxenZc/AcNpTgmTNb8pHPpaIIjv4Ml0/rcCXOfz2SS1G5ucGwlpSSTZ/sJ/1gPtfc14/Q7r71tmsNQgg8/dzw9HOja/TZBwBTmZmcOqGx3T+exGrWnvlcXHUEdvGyZYsFRXghEYgGghqGqCiC/vwQ2e/Mo/iWW/AaOdLhn6Uh2jKE1FFp0pFIKS1CiLiaK9vtQUpprgqDrUdL/10kpUwWQryAtshldVXTu4Dldfq+AxgJBAohplQdq07z/UQI0QktdJYIPGSvTQrH0tZ6JIWmQpukbrUeib2OxGoykTZ9Oq6dOxP+1lsIFwfF6g9tBIupQ65mX5+cgYdBz4ie9Y/gdq4/zv7fMhh8Yzd6DW7b0ikGowthPfwI62Er4YfFYiXvdKlt1JJzsohDCVkkb64KZHhcgk5WsuKlbbi66XF1c9Fe3fW4uulxCR5NcUwW6W+vpbM1CoO3sapdja2qratBj7BzxKNoGnv/m3YBXwshPgNsq3+klKsau0hKuRZYW+fY83X2Z9dz3X+A/zTQ59V22mw3+d8cxpTe9KKmytPaE3H1XEljGMI88Rt3SaNtlB5JbZrSIyk3lzP53sn0iuxl0yMxGo1Njmak1Yrp4EEAIhYvQu/n12DbZpO6Doz+EFFvHkm7YbVKNqRkclXv+iV1DyVk8dtXR+g5OITBN3RtewPrQa/XVYW4vOAP2jEpJUVnysk5WcwP8zZh1Rnw8nenssJMebGJolwLlRXaZiq3IP20v8v9yw83cicNF4OuhoNxwVDX6dR0PNVO65zjeoyWLiCgorRj6fsJqTlKq1XaHSZsKfY6kgAgF608SjUSaNSRKBqmph6J2WwmNjaWuLg4QNMjeeABrSbmc889x8KFC3nssceAs3okq1evZty4cWzdupWPPvqIwYMHk5iYSExMjE2P5NVXX+WWW26x6ZGkpKRwzz33MH78eJseibu7OwcPHuSuu+6qpYlSk2o9kr///e9YLBZKnSB9WlOPxMXFhWnTprF8+XIuueQScnJy2LpjKyeLTuJr9aVLcBfeffdd5s+fb1dJ+8q0NKylpXT5v/dw69nTcUZbKuHAek17RO/YbKTWsutkHtlFFfWGtTKPFbJxSQqh3X25enKfNlkD0lKEEPgEGvEJNPJr5RkAbnj4pnrbSimxmiUn/jGHvA0/0nne+4iwLpqjKT/rcLT35rP7NbaKMjPF+RVUllswVbWpDrc1RHVA8KMnNjvyo7eaEMIAKMgqxT/UsVILdbF3Zbu98yLnJU2NHKpxZNaW0iOpTVN6JDP+MoOhVw9l6m3N+1OsTE/HnJWFPigI76sdPJg98RuU53fQsFYmrnrBqD61M5iKzpSz9v/24OFj4PqHonGpZ7RyviKEQO8qiHjmL1T+tIGKd/9J1KefNL+Kcx0sZmu9TkfbzHz9n30ICdfc1MNBn8QxrF29HQCjt/Pr69q7sn0x9aTuSinvdbhFFwmNTTdNmTKFr776igEDBrBkyRI2bdpkO+cMPRKr1Yq7e8MaFdV6JGvWrGHSpEnMnDmTyZMnt+yDN4Cs0iN58cUXzzmXuDuRhZ8t5L8f/ZdtG7axYMECu/vNeV+r6uMaFuYwW22krgW9G1zi8Ghrq5BSsj45g6GXBOHjfnb1v6nczJr39mA2WbjpLwPx8Dm/CnjbG55x8fcn+JmnOf3sX8lf+Rn+d05s1X31Ljr0LroGy8OYVuwHYMDoiHrPtxeff7sJoE3K2tjrqv+Hloq7Bm0RoA9aBpeihSg9kto0pkdSYirh2vHX8o9Z/7Dd29vbu8ny+qbjx8lf9SUunTq1fuV6XaSE/Wug+5Xg1rHUplMzizieW1qrZLzVKvluYTJnTpcwdtqlBIQ5N9ThFAxe2mYHvjfdhMeQIWS99Rbm7OymL7gA8TTo8TS0zYjT3tDWFzX3hRD/BTY6xaKLBKVHUpvG9EgmT52M2WrG6GK06ZFMnTqV+++/v9HJ9uz57yFcXZ0zGsnaB/nHYcRfHN93K1m/NxMh4Jq+Zx3JL58f4nhSLlfe1YvIfoGNd2Dnl3VHpnptydGbbiLzlVcJf/ON9jbpwsaeWvN1N6A3Wh2tdtcasWdTeiTOoS30SKxWq9yfu18eLzhu9zUpKSmy/MABmdKnr8x8/fUm9UhaxE+vSTnLR8qC9BZ3MWXdFDll3RQHGqVx/ds/y1veO6urkvRTmpz/4Pfy5xWpdl3/6f2fyE/v/8ThdrWWltiVNe9dmdK7T+t0Zppg1RsJctUbHe9/919/+0T+62+t+z3iSD0SIUSREKKwegO+QStNomgFSo+kacrMZZit5mbX1sp+dz46Dw8C7ruvVfdvkNR1EB4HPp2d038LOXmmlJTThbZsrRMpufy8/ABR0YEMv92+jLVh5d8yrPxbZ5rZZgROewBD165kzJmDtby8vc25YLE3tOUYBSFFLZQeSdMUmYoQCLyaEW6RlZUUffcdQY88got/wzWmWkzhaTiVAFc/16puFl+32EEGneW7FK3k+9j+oZxJL2H9gr0EdPbk2vv6O30tQUdE5+ZG6OzZnJgyhZx//5vgv3S8UOSFgL1ZW7cAP0gpC6r2/YCrpJRfOdM4RdvSEfVICk2FeLh64KKzf52GpbAQg69v64SqGuPAOu21gxZp7B3iTYibK5+/vQO9Qc8Nj1yGwd3+n1/UMsfPgbUnnn8Ygu9NN5G7cBG+N96IW4+OlaZ7IWBv3GFWtRMBkJrAVGMl5BWKVlNhrsBkMTUrrGUpKUVWVBB4333ovZ00kE5dB/5dIbhjCSnlFlew49gZxvYJZu37SZQUmLjhz5fhHdBwavfFQvAzT6P38OD0rNlIq7XpCxTNwl5HUl+7jrWUV3HBUa090hxtdnNWJuh0BPzpj84xqqIYjvykLULsYCvCN+7LxGqFiMPlZBwpYMyUfoR0a70c8YWAS0AAwU/PpCwhgYJVF0dBjt4Bfegd0KdN7mWvI9khhHhLCHGJEKJ7ldZ6gjMN64gsXryYxYsdH9dW1E+hqRCji9FuSV1LcTHWkhL0Xl7oPDycY9ThH8BS0WFXs48VRrL2nmHI+O70iFO6HDXxvfVWjIPiyHz9Dcy5ue1tzgWFvY7kMcAErABWAmXU0Q9RKBxJpaWScnO53aMRKSXmzCyEqyuikTUxrSZ1Lbj7QeRQ592jBRRXmMnae4bL8qH3H0KJuz6qvU3qcAgh6Dx7NtbSUrJee629zbmgsMuRSClLpJTPSikHVW1/k1I2XS5XcUGj1+trpf/Gx8eTn59vS/81GAy2leqgrVaPiYkhJiaG0NBQwsPDbfsmk6lW301J6i5atIiMjAzbvrW4GGtZKS6dOjmvCKHFDAe+hV5jO1yRxvU/HeeaYhe8ungy6o8duxBje+LWoweB999HwderKfn116YvUNiFvetINlRlalXv+wsh1jvPLMX5QHWab1jVyvG1a9fi5+d3zvFqqvVIEhMTeeihh5gxY4Ztv+7K9CJTEW56twYldWs6Em00kokwGBxbIr4uJ3+HsrwOF9YqzCkj7ZsTlOrhtukx6F1bt3bnQifowQdxjYwkY/YcrBUV7W3OBYG9j1VBVZlaAEgp84QQF0wAdt26dbWebhuiuo098yShoaFcf/31jbZReiS1qdYjqTBV0De2L0qk2AwAACAASURBVHPfmYvZbLbpkUgpmTZtGiEhIbX0SH7dsAHKy3END291pddGSV0LegP0GO28ezSTijIz/3tvNxazleI/+OPl4+CaYhcgOnd3Qmc9z8n77if3gwV0mv5Ye5t03mPvf51VCBFZvSOE6Eo91YAV9lNTj2TVqlW1tEBuvfVWtm/fzu7du+nbty8LFy60navWI5k7dy7jxo1jxowZJCcnk5SUZFsDUq1HkpCQgLe3t02P5Msvv+T55zVdsWo9kp07d7JixQqmT5/eoK3VeiSJiYns3r3bKWXka+qRbPptExazhfVfrSchIYGcnBySkpLYu3cvkydPZuLEicTExLBixQp27dqFyM9HuLk5dzRSXaSx20hwc0xa8Zdv7uTLN1teANNqsfLdh3vJyyzlKw8To4eEO8SuiwGv4cPxufFGcj/8kIojR9rbnPMee0ckfwe2CCF+qtofCUxzjkltT1Mjh2qqRyJTp7ZenkXpkdSmph5JpbWS0tJSontEMy5+HKmpqTz++OPEx8dz7bXX1rrOUlCArKjAEBHh3HmB7FTIOwrDHnXePZrJls8OcSLlDHn9vMjNMTHskpaNBC9WQp59huKffyZj1mwiP16q5pVagb2T7d8Cg4BUtMytJ9EytxQtRDahRzJ//nySkpKYNWsW5TVqBDlDj2THjh3nTHbXpFqPJDw8nEmTJjml+q+s0iNJ2JnAZz9+xi+7f+H5558nMDCQPXv2MGLECObNm8eDDz549hqrFXNWFjp3d3Q+Tl4vkVpV5r+DzI/s+TGNpE1pDBgdwVeFBVzVJ7heSV1Fw7gEBRH85JOUbt9OwVdft7c55zX2Trbfj6ZD8mTVtgyYbcd11wkhUoUQh4QQz9Zzfq4QIrFqOyCEyK9x7h4hxMGq7Z4ax+OEEElVfc4T5+ljhNIjqU21HsmJ0yeQUlJZWGnTI5FSMmHCBObMmVNLj6Tg9GmkyYRLcLDznyZT10HYQPBxQkn6ZnJ8by5bVh6g62VBGOICyCk21Supq2gavwm3Yxw4kKxXX8Wcl9fe5py32BvaehwYDPwmpRwlhOgDzGnsAiGEHngPuAZIA7YLIVZLKVOq20gpZ9Ro/xgwsOp9AFoJlkFoczEJVdfmwf+3d+fxUZVXA8d/JzskBMjCIoGwiyyuuKFSEEXccGsVbZVaqVaL2mqtpda61bdWX9u+rdalaF1aS12RVjAgizsVFBRCCLKGANkmgWyEZGbO+8fcYAjZM3dmgPP9fEImN3c5kzBzcp/73HN4isCw2nJgPjAFWNDG5xExrB/Jger7kVx0/kV4fV6SEpL29yO54YYbUFVEZH8/ku9Pn86NM2cG+pGsWoWrf4tXFEL+Spj4SzeP0iaeHZVkzV5LakYS5/5gJI++t4G46CgmHp0e7tAOSRIVRZ8H7mfL5VdQ9Nj/ctT/PNzufVx2Z2RW7Q5pXG2pNQ+scD6vBuLrH7eyzelAVoOvZwGzWlj/E+Bc5/HVwDMNvveMs6wvsL7B8gPWa+7D+pG4I9j9SHx+n64rWaf55fmtrltXUqLVa9ao1/kZNtT4d9vpfiQr/xboPbJrTcf30YT29rGo2rNPX5z1sf7t5x9qRele9fv9eubvFuv05/8b1LgilSt9ZRyF//u4rjt6hFb+98j4WbYVwexHAuQ795HMBRaJyNtA0zXEv9EP2N5wH86yg4hIJjAIWNLKtv2cx63u81Bg/UgOVF1XjV/9rd7Nrn4/3uJioromEuXmXez1chdA9wHQe5T7x2qGt9bH/Ke+Ym9lLRfccixJPRPI2VXB9tK9NqwVBGm33ExsRkbg3pIWrheaprW1H8llzsP7RWQp0B1orfNNU4PWzV1hnga8rqr1A/XNbdvmfYrIjTgzywYMGNDUKmFn/UgOVF5bTpREtdp7xOfxoF4vsW7P1AKorYLNy+Ck74etSKP6lcUv5VC4tZzzbxxDr8zAxIKs7AJE4JwGLXVNx0R16UKfX9/L9htvwjN7Num33BLukA4p7a7zoKrvt74WEDhb6N/g6wyaP4uZxoG1u/KBCY22XeYsz2jLPlX1WeBZgLFjx9o9L20Qzn4kqkpFbQVJcUlESfNnMerz4S0pISopiehQnI1sWgremrDO1vrsP1vYuLKI0y8bwuATvrkWkpVdwNjMnqR3s5sQgyFp/Hi6nT8Fz9PP0P2CC4gbODDcIR0y3KylsAIYJiKDRCSOQLKY13glETka6Ak0LHyTBUx2SrH0BCYTuN6yC6gQkdOc2VrXATZv7zBQ31K3tWEtr8eD+nzE9grRX+G58yGhO2Q2fde/64f/bwEr52/lmHF9OWHyN2fWeZ5q1hdU2LBWkPWeNQuJi2PXAw+0OEXfHMi1RKKqXmAmgaSQA7yqqtki8qCITG2w6tXAHG3wW1PVUuAhAsloBfCgswzgZmA2sBHYxCE4Y8scrLy2HEHoFtt8IlGvF19JCdHJyUR17eJ+UH5foEjjsMnQxlL2wbRz426WvJxDv6N78K1rjj5gGC8rO1CuZ/JISyTBFNurF73uvIPqT5dT/u9/t2mbbddex7Zrr3M5ssjmaglTVZ1PYIpuw2W/bvT1/c1s+zzwfBPLVwKjgxdl233+xTUAnHRiZF3bONTVD2slxiUSHdX8RF5viQf1+4npFaIyb9s/g2pPWIa19hTvZcHTa0hO7cKUG8cQHXPg33xZ2QWM6NONAaku9V05gvW46ip2z51L4SO/I2n8eHdL7xwmrEyoCbt9vkBL3ZaGtbSuDm+ph+ju3YlKCFHr2Nx3ICoWhp4TmuM59lXX8c6TX6KqXHjLsSQkHng2VFyxj8/zymxYyyUSFUXfBx7At2cPRY8/Hu5wDgmWSEyHBasfydgTx1JXW9fisJa3pAT8uv9spHE/kqBThfXzYdBZkBC6drU+n593n13LnuK9nH/TGHr0PviM472cQlQ54hJJ5ssvkfly8G+GbUrCiBGkTJ/O7tdep/rzI64ZbLtZIjEdFqx+JG9/8DbJicnNttT119XhLS0lumcPopy6Ya4nkpKvoXRTSIe1VJUP52wgf30ZE747gn7Deza5XlZ2Af1TunBM3+BUITZNS5/5Y2KO6suu++5D7d6SFkVWm7cw2bDhISoqc1pdr6IiUN2l/lpJS7olHcPw4fe2uI71I4FaXy013hp6J/be34+ktraWcePG8cQTT+D3+5k+bRpfrlkDsbHceNNNB/Uj+eyzzw5qjNVp+4s0tq0ydDB8uXg72R/u5MTzMjlmXN8m16moqeOTjR6uOz3TqtW6LKprV/rcey/5N9+C528vkHbTYVPwPOjsjCRMrB9JQEVtoI5Y3oa8/f1IVq9ejdfrZc6cOaxYvpzioiJWLVvG2uzsg/qRNNVdMShyF0Df46B7RuvrBsGWr0r4+I2NDDkhndMuGdzsektzi6n1+Tlv9JE1rBUu3SZOpNvkyZT85S/U5uWFO5x2CeVsMjsjgVbPHOoFc9aW9SMJKK8tJz4mnkVLF+3vRwKBMiv9+/dn4pgxfL11K3f+9rdceNFFB/UjcUVlUWDG1oSDCla7onh7BQufy6bXgG5Mun4kEtX8mUZWdgFpSXGcOKDpYS8TfL3v+SVVH39MwYMP0f+vz9qZYBPsjCRMWrrZ6UjpR+JXP9V11STHJe/vR1J/DSU3N5d77rqLHiJ8sXQZZ40ff1A/EtdseBfQkFwf8Xn9zP/LVyR0jeGCW44lNq756c81dT6WrS/i3JG9iW4h2Zjgiu3dm/Sf/ISqjz6ifP781jc4AlkiCRPrRwI13kCC7BbXbX8/kvpZXh6Ph82rV1O8ezfRqSlN9iNxo7w+4BRp7A99xrizf4ffr5TurKKm2suFPz6WxO4tlzr5ZFMJVbU+Jh9hs7UiQc9rriZh9GgKf/sIvvLycIcTcWxoK0ysH0ng/pHYqFgSohP29yM555xz8Pv9xEZH88e77yY+NZVLzj77oH4k119/PTNmzAj+xfba6kB9rROvdbVI4+7Cajz5ldTt83HBLceSltH6DKystYUkxccwbkiqa3GZpkl0NH0euJ+t37mSot//nr733x/ukCKKHAn1ZMaOHasNL2YD5OTkcMwxx4QpooDKykqSkpKorq5m/PjxPPvss4dUKfmkpCQqKyub/X5Ls7h8fh+5Zbn0TOhJ38SDZyjVbtuGv7qa+OHDkej2ta1q/Lutv+DYpnsQ1s+HOVfDtXNhyMR2HbctVJV1H+3ko9e+xudVevTuwjX3ndbqdj6/csrD7zFuaBp/vvqEoMdl2qbwt49Q+uKLZP7zFbqeEPg9tOv/VwgFIy4R+VxVx7a2ng1thdGR3I+ksq4SVSU57uCb/fzV1fgqKohOTW13Eum03HcgPhkyzwj6rvdW1LLg6TUs+0cufQZ3p1dmN7okte1MauXWUjxVtZw3ykrGh1P6bbcS06cPBffdj9bVhTuciGFDW2F0JPcjqaitIDoqmq4xB9+5XVdUhERHE5Ma4iEcvw9y34Vh50JMcKcU52V7WPxiDjXVdYy7YijHT+rP3D+savP2WdmFxMVEMeHoENUZM02KSkykz72/Iv/HMyl98UVSZ8wId0gRwRKJ2S9U/Uj86qeitoLk+OSDplL6qqrwV1YS26dP6M9G8ldCdUlQZ2t563x8+uYmvlqaT8++iVx823Ftuh7SkKqSlV3AmUPTSIq3l2y4dZs0iaRJkyh+4km6TQndDauR7Ij+X1l/AdeEVlVdFX71HzSspap4CwuRmBiiU1I6tO9OXfPLfQeiYoJWpLEkv5JFz2dTurOKMRMzGHfZEGJamN7bnOyd5ezYvZfbJg0NSlym8/r86h42XXgRBQ89aO8jHMHXSBISEvB4PNa8JgwqaiuIkigSYw+cKeavrMRfXU1MejrSjl7v9VQVj8dDQkerA+cugIFnQpfOlQ1Xv7L6vTxee2QFeyvruGjmcYy/aniHkgjAwuwCoqylbkSJ7duX9Ntuper9D/CVlYU7nLA7Ys9IMjIyyM/Pp7i4ONyhHFFUlcLqQuKj48ktyj3ge97iYtTvJzYqCoqKOrT/hIQEMjI6UNakZCOUbICTOzfmXbV7H++9sI789WUMPDaNs68dQZdunbvekpVdyNiBKaQmWUvdSJLyve+xZ948ar/eSHRy6CpER6IjNpHExsYyaNCgcIdxxFlVtIrbFtzGo+MfZdygb4pEVrz3Hvkzb6Xvww/TY8KE0AcWhCKNm1YVsfTv6/HV+vnWNUcz6qyjOj3ksbWkitzCCu69aGSn9mOCT2Ji6PvAA2z9zpXU7dgR7nDC6ogd2jLhsXjbYmKjYjmr3zfVg9Xvp/j//kRcZibdL5nawtYuyl0QuJO9x4DW122ktsbLkpdyePeZtSSnduHKe05m9Ph+QRk3/6alrg1rRaIuY8YQ06sX3qIits+cSU0LNesOZ64mEhGZIiK5IrJRRJqsgCciV4rIOhHJFpFXnGUTRWR1g48aEbnU+d4LIrKlwfeCV0HQuEpVWZy3mFP7nkpSXNL+5eULFrDv669Ju/VWJCYMJ8lVJbD9vx2arVW4pZxXH15Bzqe7OHFKJlf8/CR69mm+SkB7ZWUXMOqoZPqnWEvdSBXbvz+xRx1F9afL2XLJpez42V3UbtsW7rBCyrVXrYhEA08C5wL5wAoRmaeq6xqsMwyYBZyhqmUi0gtAVZcCxzvrpAAbgYUNdn+Xqr7uVuzGHRvKNpBfmc8Pxvxg/zL1ein58xPEDxtG8gVhmkq54V1Qf7sSid/n5/N3t7Hina0kdo/j0p+e0Gwjqo4qKq/hi7zd3HHu8KDu1wSXREUR268fA994ndLnnqP07/+gfMECelx+GWk330xsowZvhyM3z0hOATaq6mZVrQXmAJc0WueHwJOqWgagqk1dYf02sEBVq12M1YTAkrwlCMLE/t+UHtnz9jxqt24l/fbbOjRTKyhyF0Byv0D/kTYoL9nL3N+v4rN/b2HoSb2Ydu8pQU8iAAvXFQJHXkvdQ1VMz570+tnPGLpoIT2vvpo9c99m03lTKHj4fwKtog9jbr5y+wHbG3yd7yxraDgwXEQ+FpHlIjKlif1MA/7ZaNnDIvKViPxBRGwqyyFicd5iju91PGldArW3tLaWkiefJGH0aJImTXLtuC32+q7bC5uWBC6yt3JNQ1XJXb6LOb/5DM+OSs65fiSTbxhFfNemWwR3VlZ2AQNTuzK8d1LrK5uIEZOeTp9f3cOQdxeQfMlUyl55hY3nTqbo8d/j27073OG5ws1E0tSrsvFNGzHAMGACcDUwW0T2T+IXkb7AGCCrwTazgBHAyUAKcHeTBxe5UURWishKm+IbfvkV+eSW5TJpwDcJY/cbb1C3cyfpt9/u6g1dbz3+BW893kzp+83vQ111q8NaNVV1LHwum/deyCEtI4mrfnUKR5/q3pnCnr11fLrJw3mj+hzxN7sdqmL79eOo3/yGwf/5N93OPhvP7NlsPOdcip98El9lVbjDCyo3E0k+0L/B1xlA4+JM+cDbqlqnqluAXAKJpd6VwFuqur86mqru0oB9wN8IDKEdRFWfVdWxqjo2PT09CE/HdMbivMUAnD3gbAD8NTWUPPU0XU46icQzg18gsc1y34G4boEbEZuxI7eMf/3mMzZ/UcypUwdz6R0nkpzWxdWwlq4vwutXJluRxkNe/KBB9Hv8fxk0dy5dTzuVkj8/waZzzsHz3PP4GzStO5S5mUhWAMNEZJCIxBEYoprXaJ25wEQAEUkjMNS1ucH3r6bRsJZzloIE/ky7FFjrSvQmqJbkLWF4z+H07xb426Lsn3PwFhUFro2E6y9uv98p0ngOxBw8Qurz+vn0rY3M/eMqomOjuPznJzH2goFEhaA74cJ1BaR3i+eE/tZS93CRcPRw+j/xBANfe5WEUaMoeuwxNp07mdJXXkFb6FB6KHAtkaiqF5hJYFgqB3hVVbNF5EERqb9ZIAvwiMg6YCmB2VgeABEZSOCM5v1Gu/6HiKwB1gBpwG/ceg4mODx7PawqWrV/WMtfVYXn2WdJHHc6iac0eUIZGjs+h6qiJoe1ygqqeOPRz/kiK4+R4/py5S9PpvfA0Ny9XFPnY1luMeeO7B2SpGVCq8uYMQx4bjaZL79E7IABFD74EJvOv4Ddb76FOq2wDzWuTtpX1fnA/EbLft3gsQJ3OB+Nt93KwRfnUdWzgx6ocdWy7ctQdH8iKX35ZXxlZaTffnt4A8t9ByQ6UDbeoapkf7iTj1/7mpi4aM6/aQyDTwjt0OhHX5dQXeuz2VqHua4nn0zm31+m6qOPKP7j/7Hrl7/E89e/kn7brXQ777zwzWLsgCO2RIoJncV5i+mX1I/hPYfj27MHz3PPkzRxIl2Oa9t0W9fkLoCBZ0CXwPBRdXktS/++nq1fldB/ZAqTrjuGxB6hnxSYlV1At4QYTh9sLXUPdyJC0llnkXjmmVQsWkTxn/7Ejp/eQfyIEaTffhtJEyYcEpMtDp2UZw5JlbWVLN+1nEkDJiEieF54AX9FBem33RrewDyboHj9/mGtbWs9zHnov2xfV8qZ3xnGxTOPC0sS8fr8vJdTyNkjehEXYy/PI4WIkDx5MoPffpujHv0d/qoq8m++hW3TrqZq+fJwh9cqOyMxrvpox0fU+euYNGAS3tJSyl58iW7nTyGhQU/1sMgNjLh6B0/hkzkbWLMsn5SjErnkJ6NI7Rea+zYuu/Pg1sortpZRVl1nw1pHKImOpvvUqSSffz6733yLkqeeIu/719P1tNNIv/22/X3iI439yWNctThvMSkJKRyXfhyev87GX1ND+syZ4Q4LchdQnHwOrz5VxJpl+Rx7dgbfmTU2ZEmkOVnZBcTFRPGt4TZl/UgmsbH0vOpKhmS9S+9Zv2Dfhg1su/oatt/0I2pycsId3kEskRjX1Ppq+XDHh0zsPxF/sYeyV16h+8UXEz9kSFjj0ooSVq1L5/WNN7Ovqo6LbzuOs64cTkxsiFv7No5LlUXrChk/LI1Ea6lrgKj4eFKmT2foooWk//SnVK9axZbLLif/Jz9l3+bNre8gRCyRGNcs37WcqroqJg2YhOeZZ1Cfj7SZPw5rTJVlNbz9xxV8UjGdzGHxTPv1KQwYGRkXtdfuCLTUnWzDWqaRqMRE0m66kaHvLSL15h9R+cEHbL7oYnb+Yha1+fnhDs8SiXHPkrwlJMYmcqI/g7LXXqPHFVcQ179/6xu6ZOPnRcx56DMKd0UxsdffOf+2cXRJ6lz3wmDKspa6phXRycn0uv12hr63iJTrrqN8/nw2nX8Bux54gLrCjnUVDQZLJMYVPr+PpduXMr7fePY8MxsRIe3mH4UnFp+fssJqsv66lu5pCVzV6xeMPKU7Eh3eoazGsrILOGVQCimJkZPcTGSKSUmh9y/uZsiihfS44nJ2v/Y6myZPpvB3j+ItLQ19PCE/ojkirC5eTWlNKZOjR7Nn7qP0/O41xPYJzZCN+pWibRXkrfOwba2Hwi3lAJx0fiYnD99E9L82w9GPhSSWttpcXMnXRZVcc6q11DVtF9u7N33vv5/UGTMoeeJJSl98kd3/+hc9p1+Her0haxRnicS4YnHeYuKi4hj2xhfsjYsj7cYbXT1edXkt23NK2bbWw/acUmoq60CgV2YySSnxdEmK47RLhsC//wRxSTDorNZ3GkJZ2YHeI3Z9xHREXEYGRz3yW1J/OIPiPz+B56mnITqa2D598FdXE9XV3Q6blkhM0KkqS/KWcKGOpvrdhaTOmEFMWlpQj+H3+SncUk7eukDyKM6rAKBLt1gGjEohc1Qq/Y9JoUu3uG9KyPv9gbvZh05qskhjOGVlFzCmX3f69XC3qrAJvmZ73YRB/JAhZPzxD9Ss+yHbrv8BdTt2UJufT8Jwd7tsWiIxQZdblsuOyh1cuCyRqMREUm/4QesbtUHVnn3kZXvIyy5le04p+6q9iEDvQd05deogBoxKJb1/N6S5Qoc7V0FlYYd6s7upYE8Nq7fv5meTraWuCY6EkSNJGD4cf02N60kELJEYFyzOW8zQAkj+dB0pt84kukeP1jdqgs/np2DTHvKyS9mW7cGTXwlA1+5xDDo+nQEjU+h/TAoJiW3sULi/SOPkDsXjlkXrCgBrqWuCLyohISTHsURigm5x3mJmfJpIdHdImT69XdtWlNZ8c9axvpS6Gh9RUUKfId057dLBZI5OJbVfUscK2eUugMxx0DWl/du6KCu7kMFpiQztFRktdf/1wC8AuOq+R8IcyYEsrvZZVheYZNK+V2DHWCIxQbW9fDtRX21g8HofqT+7k+iklt8cfXV+dm7aTd5aD3nrSindGWhBmtQznmFje5M5KpWMET2J69K5/6pd/TuhaB2c9z+d2k+w7amuY/lmDzPOGnxIVHk1pimWSExQLd72HtM+8CGpPen53e82uU55yV62OYkjP7cM7z4fUTHCUUN7MOL0vgwYlUJK38SgvrH29X4ceBBh10cWry/E61fOs5a65hBmicQE1abFc7kmD3rdcwtRXQIzkLy1PnZ8vXv/kNXuwmoAktMSGHFaHzJHpXLU8B7EJbj337Gv9xPoNRJSBrl2jI7Iyi6gd3I8x2V07DpSMPi8Xupqaqit2Rv4vLca9Sv56yKri/W+qsDZqsXVNnWqgc/7aoiNd/daiSWSFsx740wApl7xUZgjOVCkxvWPV0/n5Hm72ZuaBBMv5ssl28nL9rBjw258dX6iY6PoN7wHo7/Vj8xRqXTv1SUkwzkJGbeQmpMDR9/p+rHa4/VXJ7Ak5ydMO2Vom1rqqqrzpr93/xt/7V7n8b7A57qavdQ2+hxYt4a6mmrn8zdJo65mL75m2rvWj/1HGourfcpLiknt525pIlEnax3Oxo4dqytXrmz3dm/+7RJikj0uRHS4a+JNUbSppaGhgX9UIqwikCqK0HQO0fqwg0dAWvotSOAfVb/zMNJ+Xv7AZ4urbdSPt6oHF33ndWI7OHtLRD5X1bGtrefqGYmITAH+D4gGZqvqQdMaRORK4H4CL5kvVfUaZ7kPWOOslqeqU53lg4A5QArwBXCtqta68gQO/xzrkiZ+cBruH6dABP7RJGiTYUn9m35z7/uNzuQOWq0zZ3pav4vIuvivFle7qEJUFB1OIu3hWiIRkWjgSeBcIB9YISLzVHVdg3WGAbOAM1S1TER6NdjFXlU9vold/w74g6rOEZGngRuAp9x4Dvv85VQXxXP6GZFVl+nTj+8CiMi4ouO8TL7wlXCHcoAlC6/BFxXFuRfODXcoB1g4/zLiorxcPu39iHoT+vebgaHTiy59P8yRHMjiap95b5yJJOwOybHcPCM5BdioqpsBRGQOcAmwrsE6PwSeVNUyAFVtsQ6yBF5tZwPXOIteJHA240oiSexZB8CQYw9uiRpO2V/vAyI3rtQ+/cIcyYHiE32Aj9Se3cMdygESEwIn0pGURIzpCNeukYjIt4EpqjrD+fpa4FRVndlgnbnABuAMAsNf96vqu873vMBqwAs8oqpzRSQNWK6qQ511+gMLVHV0S7F09BqJMcYcqrZdex3QuVpgkXCNpKk/sxpnrRhgGDAByAA+FJHRqrobGKCqO0VkMLBERNYA5W3YZ+DgIjcCNwIMGDCgY8/AGGNMq9ycZpAPNJxzlgHsbGKdt1W1TlW3ALkEEguqutP5vBlYBpwAlAA9RCSmhX3ibPesqo5V1bHp6enBeUbGGGMO4mYiWQEME5FBIhIHTAPmNVpnLjARwBm2Gg5sFpGeIhLfYPkZwDoNjMMtBb7tbD8deNvF52CMMaYVriUSVfUCM4EsIAd4VVWzReRBEZnqrJYFeERkHYEEcZeqeoBjgJUi8qWz/JEGs73uBu4QkY1AKvCcW8/BGGNM61y9j0RV5wPzGy37dYPHCtzhfDRc5xNgTDP73ExgRpgxxpgIEGG3YhpjjDnUWCIxxhjTKZZIjDHGdIolEmOMMZ1iicQYY0ynWCIx5CQqKAAAB9ZJREFUxhjTKZZIjDHGdIolEmOMMZ1iicQYY0ynWCIxxhjTKZZIjDHGdIolEmOMMZ1iicQYY0ynWCIxxhjTKZZIjDHGdIolEmOMMZ3iamMrY4wx4ZH58kshO5adkRhjjOkUSyTGGGM6xdVEIiJTRCRXRDaKyC+aWedKEVknItki8oqz7HgR+dRZ9pWIXNVg/RdEZIuIrHY+jnfzORhjjGmZa9dIRCQaeBI4F8gHVojIPFVd12CdYcAs4AxVLRORXs63qoHrVPVrETkK+FxEslR1t/P9u1T1dbdiN8YY03ZunpGcAmxU1c2qWgvMAS5ptM4PgSdVtQxAVYuczxtU9Wvn8U6gCEh3MVZjjDEd5GYi6Qdsb/B1vrOsoeHAcBH5WESWi8iUxjsRkVOAOGBTg8UPO0NefxCR+KYOLiI3ishKEVlZXFzcuWdijDGmWW4mEmlimTb6OgYYBkwArgZmi0iP/TsQ6Qu8DFyvqn5n8SxgBHAykALc3dTBVfVZVR2rqmPT0+1kxhhj3OJmIskH+jf4OgPY2cQ6b6tqnapuAXIJJBZEJBl4B/iVqi6v30BVd2nAPuBvBIbQjDHGhImbiWQFMExEBolIHDANmNdonbnARAARSSMw1LXZWf8t4CVVfa3hBs5ZCiIiwKXAWhefgzHGmFaIauPRpiDuXOQC4I9ANPC8qj4sIg8CK1V1npMMHgemAD7gYVWdIyLfI3C2kd1gd99X1dUisoTAhXcBVgM/UtXKVuIoBrZ18GmkASUd3NZNFlf7WFztY3G1z+EaV6aqtnptwNVEcjgQkZWqOjbccTRmcbWPxdU+Flf7HOlx2Z3txhhjOsUSiTHGmE6xRNK6Z8MdQDMsrvaxuNrH4mqfIzouu0ZijDGmU+yMxBhjTKdYImkHEfmZiKhzz0vYichDTqmY1SKy0ClwGXYi8piIrHdie6thtYJwEpHvOBWl/SIS9hk2bamOHWoi8ryIFIlIRN2fJSL9RWSpiOQ4v8Pbwx0TgIgkiMhnIvKlE9cD4Y6pIRGJFpFVIvIfN49jiaSNRKQ/gUrGeeGOpYHHVPVYVT0e+A/w63AH5FgEjFbVY4ENBMraRIK1wOXAB+EOpEF17POBkcDVIjIyvFEB8AKB+7oijRe4U1WPAU4DfhwhP699wNmqehxwPDBFRE4Lc0wN3Q7kuH0QSyRt9wfg5xxcLyxsVLW8wZeJREhsqrpQVb3Ol8sJlMcJO1XNUdXccMfhaEt17JBT1Q+A0nDH0ZhTGukL53EFgTfHxkVgQ84p11R/Q3Ss8xERr0MRyQAuBGa7fSxLJG0gIlOBHar6ZbhjaUxEHhaR7cB3iZwzkoZ+ACwIdxARqC3VsU0TRGQgcALw3/BGEuAMH60m0O5ikapGRFwEqor8HPC3tmJnudbY6lAjIu8BfZr41j3AL4HJoY0ooKW4VPVtVb0HuEdEZgEzgfsiIS5nnXsIDEn8IxQxtTWuCNGW6timERFJAt4AftLojDxsVNUHHO9cC3xLREaralivMYnIRUCRqn4uIhPcPp4lEoeqntPUchEZAwwCvgyUBiMD+EJETlHVgnDF1YRXCFRLDkkiaS0uEZkOXARM0hDOMW/Hzyvc2lId2zQgIrEEksg/VPXNcMfTmKruFpFlBK4xhXuywhnAVKfeYQKQLCJ/V9XvuXEwG9pqhaquUdVeqjpQVQcSeAM4MRRJpDVOq+J6U4H14YqlIadB2d3AVFWtDnc8Eaot1bGNwynw+hyQo6q/D3c89UQkvX5Wooh0Ac4hAl6HqjpLVTOc96xpwBK3kghYIjnUPSIia0XkKwJDbxExJRJ4AugGLHKmJj8d7oAAROQyEckHTgfeEZGscMXiTEaYCWQRuHD8qqpmt7yV+0Tkn8CnwNEiki8iN4Q7JscZwLXA2c7/qdXOX9vh1hdY6rwGVxC4RuLqVNtIZHe2G2OM6RQ7IzHGGNMplkiMMcZ0iiUSY4wxnWKJxBhjTKdYIjHGGNMplkiMCQIRqWx9rRa3f11EBjuPk0TkGRHZ5FSU/UBEThWROOex3UhsIoolEmPCTERGAdGqutlZNJtA4cRhqjoK+D6Q5hR3XAxcFZZAjWmGJRJjgkgCHnNuFF0jIlc5y6NE5C/OGcZ/RGS+iHzb2ey7QH19siHAqcCvVNUP4FQIfsdZd66zvjERw06RjQmuywn0pTgOSANWiMgHBO7MHgiMAXoRuJv9eWebM4B/Oo9HAaudQoBNWQuc7ErkxnSQnZEYE1xnAv9UVZ+qFgLvE3jjPxN4TVX9Tp22pQ226QsUt2XnToKpFZFuQY7bmA6zRGJMcDVVHr6l5QB7CVRoBcgGjhORll6b8UBNB2IzxhWWSIwJrg+Aq5xmR+nAeOAz4CPgCudaSW9gQoNtcoChAKq6CVgJPOBUvEVEhonIJc7jVKBYVetC9YSMaY0lEmOC6y3gK+BLYAnwc2co6w0CLQjWAs8Q6O63x9nmHQ5MLDMINOfaKCJrgL/yTa+SicB8d5+CMe1j1X+NCRERSVLVSues4jPgDFUtcPpYLHW+bu4ie/0+3gRmRVDveWNs1pYxIfQfpwlSHPBQfXM0Vd0rIvcR6Nme19zGTgOsuZZETKSxMxJjjDGdYtdIjDHGdIolEmOMMZ1iicQYY0ynWCIxxhjTKZZIjDHGdIolEmOMMZ3y/8TczODxhxzdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(C_s)\n",
    "n_gamma_s = len(gamma_s)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,n_gamma_s)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,n_gamma_s)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,n_gamma_s)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,n_gamma_s)\n",
    "\n",
    "x_axis = np.log10(C_s)\n",
    "for i, value in enumerate(gamma_s):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    pyplot.errorbar(x_axis, test_scores[:,i], yerr=test_stds[:,i] ,label = 'gamma_s[i]' +' Test')\n",
    "    #pyplot.errorbar(x_axis, train_scores[:,i], yerr=train_stds[:,i] ,label = 'gamma_s[i]' +' Train')\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'accuracy' )\n",
    "pyplot.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7687296416938111\n",
      "{'C': 10.0, 'gamma': 0.01}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7727272727272727\n"
     ]
    }
   ],
   "source": [
    "best_rbfSVM_accuracy = SVC_performance(10.0, 0.01, X_train, y_train, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两个参数组合比较多，所以图看起来比较乱，但可以直接得出最优参数  \n",
    "最终得到的最优参数为 C=10.0，gamma=0.01  \n",
    "在测试集上的正确率为0.7727272727272727，比线性核的最优参数所训练出来的模型在测试集（0.7662337662337663）上的正确率高"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据整理与对比"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1、根据api调用，其实class_weight是有自己的备选参数的，{dict, ‘balanced’，None}，也可以根据自己的比例计算，手动设置参数，balanced参数应该是模型自己计算并进行自适应，因为本数据集的样本分类不均衡，尝试过balanced参数，并进行结果对比，发现  \n",
    "\n",
    "分类的效果 ： 手动设置比例 > 设置为balanced > 不设置参数   \n",
    "\n",
    "故本作业全部使用手动设置的参数  \n",
    "注：根据计算的比例[0:0.3489, 1:0.6511]，在训练集和测试集中跑的效果都不错，比例改为[0:0.4 , 1:0.6]，在训练集上效果更好，但测试集上效果差点，最终选择第一个比例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2、三头肌皮肤褶层厚度和血清胰岛素两个特征的0值太多了，物理含以上我认为应该去掉三头肌皮肤的特征，保留血清胰岛素特征，但实际操作中还是进行对比，并查看训练效果再决定（Ps.四种情况我都跑了下测试集，理论上测试集只能用于确定出来的模型最终跑一次看性能）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1)保留 Insulin 和 SkinThickness 两个特征"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "default LinearSVC( C=1 )--train accuracy: 0.7524658953622853  \n",
    "default LinearSVC( C=1 )--test accuracy: 0.7597402597402597  \n",
    "\n",
    "best LinearSVC( C=0.01 )--train accuracy: 0.758957654723127  \n",
    "best LinearSVC( C=0.01 )--test accuracy:  0.7597402597402597\n",
    "\n",
    "best rbfSVC( C=1000.0,gamma=0.01 )--train accuracy: 0.7671009771986971  \n",
    "best rbfSVC( C=1000.0,gamma=0.01 )--test accuracy: 0.7662337662337663  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2)保留 Insulin 特征，去除 SkinThickness 特征 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "default LinearSVC( C=1 )--train accuracy: 0.7573310460761932    \n",
    "default LinearSVC( C=1 )--test accuracy: 0.7532467532467533   \n",
    "\n",
    "best LinearSVC( C=0.01 )--train accuracy: 0.7638436482084691  \n",
    "best LinearSVC( C=0.01 )--test accuracy: 0.7597402597402597  \n",
    "\n",
    "best rbfSVC( C=10.0, gamma=0.01 )--train accuracy: 0.7654723127035831   \n",
    "best rbfSVC( C=10.0, gamma=0.01 )--test accuracy: 0.7662337662337663  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3)保留 SkinThickness 特征，去除 Insulin 特征 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "default LinearSVC(C=1)--train accuracy: 0.7589568473685795  \n",
    "default LinearSVC(C=1)--test accuracy: 0.7532467532467533  \n",
    "\n",
    "best LinearSVC( C=0.01 )--train accuracy: 0.762214983713355  \n",
    "best LinearSVC( C=0.01 )--test accuracy: 0.7597402597402597  \n",
    "\n",
    "best rbfSVC( C=0.1, gamma=0.01 )--train accuracy: 0.7671009771986971\n",
    "best rbfSVC( C=0.1, gamma=0.01 )--test accuracy: 0.7012987012987013"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4)去除Insulin 和 SkinThickness 两个特征 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "default LinearSVC(C=1)--train accuracy: 0.7605961916308746  \n",
    "default LinearSVC(C=1)--test accuracy: 0.7597402597402597\n",
    "\n",
    "best LinearSVC( C=0.01 )--train accuracy: 0.7671009771986971  \n",
    "best LinearSVC( C=0.01 )--test accuracy: 0.7662337662337663  \n",
    "\n",
    "best rbfSVC( C=10.0, gamma=0.01 )--train accuracy: 0.7687296416938111  \n",
    "best rbfSVC( C=10.0, gamma=0.01 )--test accuracy: 0.7727272727272727"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从以上结果来看，去除Insulin 和 SkinThickness 两个特征，不管在线性核还是RBF核，不管在训练集还是测试集，基本上都比其他的训练效果好，故应去除这两个特征。另外可看出，RBF核训练处的模型比线性核的训练模型预测效果更好，而对比logistic回归可发现，SVM训练的模型预测效果更好一些。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
